Nov 01 22:20:47.115694 ip-10-0-129-77 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 01 22:20:47.115703 ip-10-0-129-77 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 01 22:20:47.115710 ip-10-0-129-77 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 01 22:20:47.115953 ip-10-0-129-77 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 01 22:20:58.299703 ip-10-0-129-77 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 01 22:20:58.299723 ip-10-0-129-77 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ca26c90ca004468b8b791156941d4e46 --
Nov 01 22:22:54.805294 ip-10-0-129-77 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 22:22:55.221228 ip-10-0-129-77 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:55.221228 ip-10-0-129-77 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 22:22:55.221228 ip-10-0-129-77 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:55.221228 ip-10-0-129-77 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:55.225876 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225802    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:55.225876 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225875    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:55.225876 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225879    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225882    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225885    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225887    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225891    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225894    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225897    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225899    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225902    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225904    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225907    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225910    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225912    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225915    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225917    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225920    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225923    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225926    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225928    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225931    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:55.225970 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225933    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225936    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225939    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225941    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225945    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225947    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225950    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225952    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225955    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225958    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225960    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225963    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225965    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225968    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225971    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225974    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225976    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225979    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225982    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:55.226447 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225985    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225987    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225990    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225994    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.225998    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226002    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226005    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226008    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226010    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226014    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226018    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226022    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226025    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226028    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226031    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226033    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226036    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226052    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226057    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:55.226897 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226061    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226065    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226070    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226074    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226078    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226080    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226083    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226086    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226089    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226091    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226094    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226096    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226099    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226102    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226104    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226107    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226109    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226112    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226115    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226117    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:55.227358 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226120    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226122    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226125    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226127    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226130    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226132    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226135    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226137    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226140    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226142    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226145    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226147    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226151    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226154    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226158    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226160    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226163    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226165    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226168    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:55.227832 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226171    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226173    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226176    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226179    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226181    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226185    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226187    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226934    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226941    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226943    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226946    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226949    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226952    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226954    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226957    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226960    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226962    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226966    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226968    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226971    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:55.228281 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226974    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226976    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226979    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226981    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226984    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226986    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226989    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226992    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226994    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226997    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.226999    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227002    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227005    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227008    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227010    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227013    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227015    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227018    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227021    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227023    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:55.228760 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227026    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227028    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227031    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227033    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227036    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227039    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227057    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227077    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227081    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227084    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227087    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227090    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227093    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227096    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227098    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227102    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227104    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227107    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227109    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:55.229242 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227116    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227120    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227123    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227126    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227128    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227131    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227136    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227139    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227143    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227146    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227149    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227151    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227153    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227156    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227158    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227161    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227163    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227166    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227169    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:55.229693 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227172    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227174    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227177    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227180    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227182    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227184    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227187    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227190    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227192    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227195    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227197    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227200    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227203    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227205    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227208    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227211    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227214    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227217    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227219    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227222    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:55.230160 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227224    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227227    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227229    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227232    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227234    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227237    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227239    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227242    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227245    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227247    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227252    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227254    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227258    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227261    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227263    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227731    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227735    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227738    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227741    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227744    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:55.230628 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227747    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227750    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227752    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227755    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227758    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227761    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227764    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227766    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227769    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227773    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227777    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227780    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227783    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227785    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227788    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227790    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227793    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227796    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227798    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:55.231111 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227801    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227803    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227806    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227810    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227813    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227816    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227819    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227821    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227823    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227826    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227828    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227831    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227834    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227838    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227840    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227843    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227846    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227848    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227851    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227853    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:55.231567 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227856    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227859    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227861    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227864    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227866    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227868    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227871    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227874    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227876    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227879    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227881    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227884    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227886    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227888    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227891    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227895    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227898    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227900    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227903    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:55.232102 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227907    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227911    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227914    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227917    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227920    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227923    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227929    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227931    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227934    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227937    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227939    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227942    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227944    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227947    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227950    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227952    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227955    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227957    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227960    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:55.232556 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227962    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227965    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227967    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227970    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227972    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227975    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227978    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227980    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227983    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227987    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227990    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227992    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227995    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.227997    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228000    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228003    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228005    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228008    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228010    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:55.233007 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228014    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228017    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228020    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228023    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.228025    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.228979    2583 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.228990    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.228997    2583 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229002    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229007    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229010    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229015    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229019    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229023    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229026    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229029    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229033    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229036    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229050    2583 flags.go:64] FLAG: --cgroup-root=""
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229053    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229056    2583 flags.go:64] FLAG: --client-ca-file=""
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229059    2583 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229063    2583 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229067    2583 flags.go:64] FLAG: --cluster-domain=""
Nov 01 22:22:55.233507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229070    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229073    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229077    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229080    2583 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229084    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229087    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229090    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229094    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229099    2583 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229102    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229105    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229108    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229111    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229115    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229118    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229121    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229124    2583 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229127    2583 flags.go:64] FLAG: --enable-server="true"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229130    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229136    2583 flags.go:64] FLAG: --event-burst="100"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229139    2583 flags.go:64] FLAG: --event-qps="50"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229142    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229145    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229148    2583 flags.go:64] FLAG: --eviction-hard=""
Nov 01 22:22:55.234088 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229152    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229155    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229158    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229161    2583 flags.go:64] FLAG: --eviction-soft=""
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229164    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229167    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229170    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229173    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229176    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229179    2583 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229181    2583 flags.go:64] FLAG: --feature-gates=""
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229185    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229188    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229191    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229195    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229198    2583 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229202    2583 flags.go:64] FLAG: --help="false"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229205    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229208    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229212    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229214    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229218    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229222    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 22:22:55.234794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229225    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229228    2583 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229231    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229234    2583 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229237    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229241    2583 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229243    2583 flags.go:64] FLAG: --kube-reserved=""
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229247    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229249    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229253    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229256    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229259    2583 flags.go:64] FLAG: --lock-file=""
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229262    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229265    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229268    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229273    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229276    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229292    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229296    2583 flags.go:64] FLAG: --logging-format="text"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229300    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229303    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229306    2583 flags.go:64] FLAG: --manifest-url=""
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229309    2583 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229313    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229317    2583 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 22:22:55.235355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229321    2583 flags.go:64] FLAG: --max-pods="110"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229324    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229327    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229330    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229333    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229336    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229339    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229342    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229350    2583 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229354    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229359    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229384    2583 flags.go:64] FLAG: --pod-cidr=""
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229405    2583 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229410    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229415    2583 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229419    2583 flags.go:64] FLAG: --port="10250"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229423    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229426    2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-02843da5daff69353"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229430    2583 flags.go:64] FLAG: --qos-reserved=""
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229435    2583 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229438    2583 flags.go:64] FLAG: --register-node="true"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229441    2583 flags.go:64] FLAG: --register-with-taints=""
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229446    2583 flags.go:64] FLAG: --registry-burst="10"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229449    2583 flags.go:64] FLAG: --registry-qps="5"
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229454    2583 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229457    2583 flags.go:64] FLAG: --reserved-memory=""
Nov 01 22:22:55.235938 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229462    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229465    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229469    2583 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229472    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229475    2583 flags.go:64] FLAG: --runonce="false"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229478    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229481    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229485    2583 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229488    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229491    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229494    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229498    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229501    2583 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229504    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229508    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229511    2583 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229514    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229517    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229521    2583 flags.go:64] FLAG: --system-cgroups=""
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229524    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229532    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229535    2583 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229538    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229543    2583 flags.go:64] FLAG: --tls-min-version=""
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229547    2583 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 22:22:55.236563 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229550    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229553    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229556    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229559    2583 flags.go:64] FLAG: --v="2"
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229563    2583 flags.go:64] FLAG: --version="false"
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229567    2583 flags.go:64] FLAG: --vmodule=""
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229572    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.229575    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230398    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230405    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230409    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230413    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230416    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230419    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230422    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230425    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230428    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230431    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230434    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230436    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230439    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:55.237159 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230442    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230444    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230447    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230450    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230453    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230456    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230459    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230462    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230464    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230467    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230470    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230473    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230475    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230478    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230480    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230483    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230486    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230488    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230493    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:55.237655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230498    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230501    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230503    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230506    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230509    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230512    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230514    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230517    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230519    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230522    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230525    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230528    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230530    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230533    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230535    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230538    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230540    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230545    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230548    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230551    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:55.238162 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230554    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230556    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230559    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230562    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230564    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230567    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230570    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230572    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230575    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230578    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230580    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230583    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230586    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230589    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230591    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230594    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230597    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230602    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230605    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:55.238653 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230609    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230612    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230615    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230618    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230620    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230623    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230627    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230630    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230632    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230636    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230640    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230643    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230646    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230648    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230651    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230653    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230656    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230658    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230661    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:55.239118 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230664    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230666    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230669    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230671    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230674    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230677    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230679    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230682    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230685    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230688    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230690    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230693    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230696    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230698    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230701    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230703    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:55.239574 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.230732    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230770    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230776    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230779    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230784    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230788    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230792    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230797    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230800    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230803    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230805    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230808    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230811    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230813    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230816    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230819    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230821    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230824    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230827    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230829    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:55.239949 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230832    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230834    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230837    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230839    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230842    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230845    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230848    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230850    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230853    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230855    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230858    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230860    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230863    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230865    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230868    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230870    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230874    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230877    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230881    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:55.240830 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230886    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230889    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230892    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230895    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230897    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230900    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230903    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230905    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230908    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230910    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230913    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230916    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230919    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230921    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230924    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230927    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230929    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230932    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230935    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:55.241333 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230937    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230940    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230943    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230945    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230948    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230951    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230953    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230955    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230958    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230961    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230965    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230968    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230971    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230974    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230977    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230980    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230982    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230985    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230987    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230990    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:55.241790 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230992    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230995    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.230997    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231000    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231002    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231005    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231008    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231010    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231013    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231015    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231018    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231020    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231023    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231026    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231028    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231031    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231033    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231036    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231038    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231053    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:55.242300 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231056    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231059    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231063    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231066    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231068    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231072    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231075    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231078    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.231080    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.237722    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.237737    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.237752    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.237757    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237818    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237824    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237827    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237830    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237833    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237835    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:55.242773 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237838    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237841    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237843    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237846    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237848    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237851    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237853    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237856    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237859    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237862    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237864    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237867    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237869    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237872    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237875    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237879    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237882    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237885    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237889    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:55.243251 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237893    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237897    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237900    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237902    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237905    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237909    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237911    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237914    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237917    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237919    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237922    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237924    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237927    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237929    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237932    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237935    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237937    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237940    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237943    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:55.243720 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237945    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237948    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237950    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237953    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237955    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237958    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237961    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237963    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237966    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237969    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237971    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237974    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237976    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237979    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237981    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237984    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237987    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237990    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237993    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237996    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:55.244201 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.237998    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238001    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238004    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238006    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238009    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238011    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238014    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238017    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238019    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238021    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238024    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238027    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238029    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238032    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238034    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238037    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238052    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238056    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238058    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238061    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:55.244675 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238063    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238066    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238069    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238071    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238074    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238077    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238080    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238083    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238085    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238088    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238091    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238093    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238096    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238099    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238102    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238104    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238107    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238109    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238112    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:55.245142 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238114    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238117    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238119    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.238146    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238293    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238298    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238301    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238304    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238307    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238310    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238312    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238315    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238317    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238320    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238323    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238325    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:55.245597 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238327    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238330    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238332    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238335    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238338    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238340    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238343    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238346    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238349    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238351    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238354    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238357    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238359    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238362    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238365    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238367    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238370    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238372    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238375    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238377    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:55.245979 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238380    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238382    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238385    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238387    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238390    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238392    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238394    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238397    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238399    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238402    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238404    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238407    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238409    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238412    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238414    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238417    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238419    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238421    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238424    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:55.246470 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238427    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238429    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238432    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238434    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238437    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238440    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238442    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238445    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238447    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238450    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238453    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238455    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238458    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238460    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238462    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238465    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238467    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238470    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238472    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238475    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:55.246944 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238477    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238480    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238483    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238485    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238488    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238490    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238493    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238495    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238497    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238500    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238502    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238506    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238510    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238513    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238515    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238519    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238523    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238526    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238529    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:55.247599 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238532    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238535    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238537    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238539    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238542    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238544    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238547    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238549    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238552    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238554    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238556    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238559    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238562    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238564    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238567    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238569    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:55.248058 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.238591    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238621    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238625    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238628    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238631    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238634    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238636    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238639    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238642    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238644    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238647    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238649    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238652    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238654    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238657    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238660    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238663    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238666    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238668    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:55.248432 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238671    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238673    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238676    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238678    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238681    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238683    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238686    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238688    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238691    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238693    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238696    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238698    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238701    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238703    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238706    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238708    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238711    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238713    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238716    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238718    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:55.248869 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238724    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238726    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238729    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238731    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238733    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238736    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238739    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238741    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238744    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238747    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238750    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238755    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238758    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238760    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238763    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238766    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238768    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238771    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238773    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:55.249346 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238777    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238781    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238784    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238787    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238792    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238796    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238799    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238802    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238805    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238807    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238810    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238812    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238815    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238819    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238822    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238824    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238827    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238829    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238832    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:55.249811 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238834    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238837    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238840    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238842    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238845    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238849    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238852    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238855    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238857    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238860    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238863    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238865    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238868    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238870    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238873    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238875    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238878    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238880    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238882    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238885    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:55.250336 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238887    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238890    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238892    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238895    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238899    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238902    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238906    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238908    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238910    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:55.238914    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.238961    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.243352    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.244169    2583 server.go:1014] "Starting client certificate rotation"
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.244260    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.244296    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.246934    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:22:55.250837 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.250201    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:22:55.262253 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.262236    2583 log.go:25] "Validated CRI v1 runtime API"
Nov 01 22:22:55.264716 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.264702    2583 log.go:25] "Validated CRI v1 image API"
Nov 01 22:22:55.265677 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.265657    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 22:22:55.270336 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.270310    2583 fs.go:135] Filesystem UUIDs: map[57d88cbb-d85f-4a69-b760-ef0af37efad2:/dev/nvme0n1p3 5f1f435a-d8d8-40df-837b-767ead34068d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 01 22:22:55.270422 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.270336    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 22:22:55.275509 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.275399    2583 manager.go:217] Machine: {Timestamp:2026-11-01 22:22:55.274033658 +0000 UTC m=+0.353033619 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102680 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27b2bf15ce1934842400f14ee02186 SystemUUID:ec27b2bf-15ce-1934-8424-00f14ee02186 BootID:ca26c90c-a004-468b-8b79-1156941d4e46 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fe:ec:58:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fe:ec:58:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:72:2b:6b:f8:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 22:22:55.275509 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.275506    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 22:22:55.275688 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.275599    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 22:22:55.278547 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.278520    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 22:22:55.278676 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.278547    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-77.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 22:22:55.278717 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.278683    2583 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 22:22:55.278717 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.278691    2583 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 22:22:55.278776 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.278725    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:22:55.278776 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.278742    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:22:55.279613 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.279599    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 22:22:55.282719 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.282708    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 22:22:55.282839 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.282830    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 22:22:55.283540 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.283530    2583 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 22:22:55.283572 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.283543    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 22:22:55.283572 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.283553    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 22:22:55.283572 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.283561    2583 kubelet.go:405] "Adding apiserver pod source"
Nov 01 22:22:55.283572 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.283569    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 22:22:55.283958 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.283942    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:22:55.290888 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.290862    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 22:22:55.296440 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.296424    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:22:55.297441 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.297429    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 22:22:55.297493 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.297459    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 22:22:55.297631 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.297621    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 22:22:55.297660 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.297638    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 22:22:55.297660 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.297644    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 22:22:55.297660 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.297649    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 22:22:55.297660 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.297655    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 22:22:55.297660 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.297661    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 22:22:55.298209 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.298200    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 22:22:55.298247 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.298211    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 22:22:55.298247 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.298218    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 22:22:55.298247 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.298224    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 22:22:55.298247 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.298234    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 22:22:55.298247 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.298242    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 22:22:55.298835 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.298825    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 22:22:55.298865 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.298838    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 22:22:55.302518 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.302505    2583 server.go:1263] "Started kubelet"
Nov 01 22:22:55.302595 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.302553    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 22:22:55.302645 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.302600    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 22:22:55.302701 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.302672    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 22:22:55.303460 ip-10-0-129-77 systemd[1]: Started Kubernetes Kubelet.
Nov 01 22:22:55.303989 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.303842    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 22:22:55.304434 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.304397    2583 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 22:22:55.309194 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.309122    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 22:22:55.313658 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.313641    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 22:22:55.314174 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.314156    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 22:22:55.314314 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.314295    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:55.314954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.314938    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 22:22:55.314954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.314939    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 22:22:55.315098 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.314963    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 22:22:55.315151 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.315119    2583 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 22:22:55.315151 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.315126    2583 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 22:22:55.315750 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.315733    2583 factory.go:55] Registering systemd factory
Nov 01 22:22:55.315835 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.315766    2583 factory.go:223] Registration of the systemd container factory successfully
Nov 01 22:22:55.316004 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.315991    2583 factory.go:153] Registering CRI-O factory
Nov 01 22:22:55.316063 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.316008    2583 factory.go:223] Registration of the crio container factory successfully
Nov 01 22:22:55.316114 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.316079    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 22:22:55.316169 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.316126    2583 factory.go:103] Registering Raw factory
Nov 01 22:22:55.316169 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.316137    2583 manager.go:1199] Started watching for new ooms in manager
Nov 01 22:22:55.317199 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.317182    2583 manager.go:319] Starting recovery of all containers
Nov 01 22:22:55.323247 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.323220    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 22:22:55.327604 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.327589    2583 manager.go:324] Recovery completed
Nov 01 22:22:55.330471 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.330430    2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 01 22:22:55.334865 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.334771    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:55.336706 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.336690    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:55.336778 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.336724    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:55.336778 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.336739    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:55.337321 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.337306    2583 cpu_manager.go:226] "Starting" policy="none"
Nov 01 22:22:55.337321 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.337319    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 22:22:55.337427 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.337339    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 22:22:55.337427 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.337351    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:55.338134 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.338115    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:55.338210 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.338147    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:55.338210 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.338161    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:55.338675 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.338657    2583 policy_none.go:50] "Start"
Nov 01 22:22:55.338675 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.338671    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 22:22:55.338788 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.338685    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 22:22:55.339780 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.339768    2583 policy_none.go:44] "Start"
Nov 01 22:22:55.365253 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.365216    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 22:22:55.367839 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.367825    2583 manager.go:341] "Starting Device Plugin manager"
Nov 01 22:22:55.367909 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.367859    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 22:22:55.367909 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.367870    2583 server.go:87] "Starting device plugin registration server"
Nov 01 22:22:55.368139 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.368126    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 22:22:55.368195 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.368142    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 22:22:55.368279 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.368227    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 22:22:55.368391 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.368365    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 22:22:55.368391 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.368376    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 22:22:55.368858 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.368841    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 22:22:55.368925 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.368889    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:55.438592 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.438569    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-77.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 22:22:55.438592 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.438582    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-77.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 22:22:55.439072 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.439055    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 22:22:55.439125 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.439077    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 22:22:55.439125 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.439098    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 22:22:55.439212 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.439134    2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 22:22:55.439542 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.438571    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-77.us-west-2.compute.internal.18e40c06e2122f2e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-77.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-77.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:55.302479662 +0000 UTC m=+0.381479623,LastTimestamp:2026-11-01 22:22:55.302479662 +0000 UTC m=+0.381479623,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-77.us-west-2.compute.internal,}"
Nov 01 22:22:55.468740 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.468720    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:55.469785 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.469763    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:55.469871 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.469802    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:55.469871 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.469815    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:55.469871 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.469845    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.486799 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.486665    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-77.us-west-2.compute.internal.18e40c06e41c7104  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-77.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-77.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-77.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:55.336706308 +0000 UTC m=+0.415706274,LastTimestamp:2026-11-01 22:22:55.336706308 +0000 UTC m=+0.415706274,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-77.us-west-2.compute.internal,}"
Nov 01 22:22:55.539454 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.539426    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"]
Nov 01 22:22:55.539543 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.539484    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:55.543767 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.543749    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:55.543845 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.543785    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:55.543845 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.543800    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:55.545222 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.545210    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:55.545371 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.545358    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.545413 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.545386    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:55.545967 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.545949    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:55.545967 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.545963    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:55.546105 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.545977    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:55.546105 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.545987    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:55.546105 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.545987    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:55.546105 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.546076    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:55.547326 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.547310    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.547407 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.547341    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:55.548077 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.548064    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:55.548172 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.548088    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:55.548172 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.548100    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:55.574755 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.574734    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.578502 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.578484    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.587089 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.587070    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 22:22:55.587207 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.587110    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-24s4n"
Nov 01 22:22:55.617214 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.617188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.617214 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.617216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.617377 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.617235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dcabca36fe7386806bacef405dbcfa5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"0dcabca36fe7386806bacef405dbcfa5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.631236 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.631211    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 22:22:55.631236 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.631215    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 01 22:22:55.631397 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.631298    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-77.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 22:22:55.631477 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.631461    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 22:22:55.631661 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.631583    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-77.us-west-2.compute.internal.18e40c06e41ccea1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-77.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-77.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-77.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:55.336730273 +0000 UTC m=+0.415730234,LastTimestamp:2026-11-01 22:22:55.336730273 +0000 UTC m=+0.415730234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-77.us-west-2.compute.internal,}"
Nov 01 22:22:55.717892 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.717870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.717976 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.717894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.717976 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.717918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dcabca36fe7386806bacef405dbcfa5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"0dcabca36fe7386806bacef405dbcfa5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.717976 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.717947    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dcabca36fe7386806bacef405dbcfa5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"0dcabca36fe7386806bacef405dbcfa5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.717976 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.717958    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.718126 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.717972    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.776812 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.776743    2583 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-77.us-west-2.compute.internal\" not found" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.824084 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:55.824038    2583 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-77.us-west-2.compute.internal\" not found" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.878348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.878327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.883574 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.883447    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:55.977187 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.977166    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:55.978068 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.978038    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:55.978137 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.978084    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:55.978137 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.978094    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:55.978137 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:55.978120    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:56.035300 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.035251    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:56.035300 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.035273    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-77.us-west-2.compute.internal\": node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.145870 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.145846    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.244665 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.244643    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 22:22:56.245039 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.244773    2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a40c4aa6d7453489f8ffaf27c6ab6881-5876fe58ba2f5954.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-77.us-west-2.compute.internal.18e40c06e41ccea1\": read tcp 10.0.129.77:36782->18.219.135.16:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-77.us-west-2.compute.internal.18e40c06e41ccea1  default   4879 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-77.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-77.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-77.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:55 +0000 UTC,LastTimestamp:2026-11-01 22:22:55.469808777 +0000 UTC m=+0.548808742,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-77.us-west-2.compute.internal,}"
Nov 01 22:22:56.246984 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.246962    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.349989 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.348175    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.449037 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.449015    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.459974 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.459935    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:56.461220 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.461203    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:56.461312 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.461238    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:56.461312 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.461254    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:56.550335 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.550312    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.591362 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.591331    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:56 +0000 UTC" deadline="2028-07-31 04:53:40.509212796 +0000 UTC"
Nov 01 22:22:56.591362 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.591360    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15294h30m43.917856096s"
Nov 01 22:22:56.651722 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.651668    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.725856 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.725833    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:56.752457 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:56.752188    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.790946 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.790926    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:56.815729 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.815711    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:56.816344 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.816321    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:56.882636 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.882617    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:22:56.883734 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.883721    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:22:56.949148 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:56.949114    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:22:57.041028 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.040977    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcabca36fe7386806bacef405dbcfa5.slice/crio-8b67db994940be8f8db241632ef45eda61a359574bd4cc5631ee23d585a73aa2 WatchSource:0}: Error finding container 8b67db994940be8f8db241632ef45eda61a359574bd4cc5631ee23d585a73aa2: Status 404 returned error can't find the container with id 8b67db994940be8f8db241632ef45eda61a359574bd4cc5631ee23d585a73aa2
Nov 01 22:22:57.041381 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.041359    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a42aef018761dafd58eb8c13dacd2c.slice/crio-1785ae234ce8733a047c2ff292d6ed523f1fa5520de77818602bb9dc8625c43b WatchSource:0}: Error finding container 1785ae234ce8733a047c2ff292d6ed523f1fa5520de77818602bb9dc8625c43b: Status 404 returned error can't find the container with id 1785ae234ce8733a047c2ff292d6ed523f1fa5520de77818602bb9dc8625c43b
Nov 01 22:22:57.045710 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.045695    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:22:57.184577 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.184554    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:57.287075 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.287007    2583 apiserver.go:52] "Watching apiserver"
Nov 01 22:22:57.316388 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.316355    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 22:22:57.401804 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.401782    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 22:22:57.402113 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.402094    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-j8gn5","kube-system/konnectivity-agent-hnmh2","openshift-cluster-node-tuning-operator/tuned-8g2k7","kube-system/global-pull-secret-syncer-njrpj","openshift-dns/node-resolver-njpz8","openshift-multus/multus-additional-cni-plugins-zd476","openshift-multus/network-metrics-daemon-xvhpg","kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb","openshift-image-registry/node-ca-km7mv","openshift-network-operator/iptables-alerter-tb9n7","openshift-ovn-kubernetes/ovnkube-node-n2phj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal","openshift-multus/multus-r5xnp"]
Nov 01 22:22:57.404895 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.404877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.406134 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.406102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:22:57.407426 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.407409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.408940 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.408927    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:22:57.409003 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.408985    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:22:57.410286 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.410270    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.411690 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.411665    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.413082 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.413064    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.413171 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.413135    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:22:57.414532 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.414513    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:57.414598 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.414579    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:22:57.415854 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.415835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.417169 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.417152    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.418767 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.418743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.420139 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.420107    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.427636 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.427729 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:57.427729 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0df7fd-8362-4c2a-8258-e79901e6c402-tmp-dir\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.427834 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-cnibin\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.427834 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427800    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.427931 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-etc-tuned\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.427931 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ee57462-5ba3-4cfe-a233-13c046a7e797-konnectivity-ca\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:22:57.427931 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:22:57.428045 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/244d4caf-546f-46aa-bdf2-8023694c22a1-iptables-alerter-script\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.428045 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.427990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.428045 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.428182 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-run\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428182 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-device-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.428182 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tds8r\" (UniqueName: \"kubernetes.io/projected/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-kube-api-access-tds8r\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.428282 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysconfig\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428282 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-host\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428282 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428257    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/a9038753-40e0-4e1f-aacc-da2a573be477-kube-api-access-mds7f\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428282 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428273    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.428479 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wjk\" (UniqueName: \"kubernetes.io/projected/244d4caf-546f-46aa-bdf2-8023694c22a1-kube-api-access-m7wjk\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.428479 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb0df7fd-8362-4c2a-8258-e79901e6c402-hosts-file\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.428479 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.428479 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.428479 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-modprobe-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428666 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-systemd\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428666 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-var-lib-kubelet\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428666 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/244d4caf-546f-46aa-bdf2-8023694c22a1-host-slash\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.428666 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-sys\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428666 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-lib-modules\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428666 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.428666 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.428666 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6jq\" (UniqueName: \"kubernetes.io/projected/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kube-api-access-nn6jq\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-host\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-system-cni-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m46p\" (UniqueName: \"kubernetes.io/projected/28838384-0663-4a33-9903-26afb7288d9e-kube-api-access-6m46p\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-kubernetes\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ee57462-5ba3-4cfe-a233-13c046a7e797-agent-certs\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-serviceca\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428927    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-dbus\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.428984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfc7\" (UniqueName: \"kubernetes.io/projected/990ac23b-9dab-47c9-a8bf-13907483a733-kube-api-access-nhfc7\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:57.429355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.428988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqct\" (UniqueName: \"kubernetes.io/projected/bb0df7fd-8362-4c2a-8258-e79901e6c402-kube-api-access-tvqct\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.429355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.429009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-kubelet-config\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.429355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.429072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.429355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.429103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-conf\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.429355 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.429128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-tmp\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.443141 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.443098    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"0dcabca36fe7386806bacef405dbcfa5","Type":"ContainerStarted","Data":"8b67db994940be8f8db241632ef45eda61a359574bd4cc5631ee23d585a73aa2"}
Nov 01 22:22:57.444097 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.444072    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"d0a42aef018761dafd58eb8c13dacd2c","Type":"ContainerStarted","Data":"1785ae234ce8733a047c2ff292d6ed523f1fa5520de77818602bb9dc8625c43b"}
Nov 01 22:22:57.472784 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472761    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 22:22:57.472784 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472760    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:22:57.472936 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472761    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 22:22:57.472936 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472760    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 22:22:57.472936 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472832    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 22:22:57.472936 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472933    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 22:22:57.473140 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472938    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 22:22:57.473140 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472954    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 22:22:57.473140 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472969    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 22:22:57.473140 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.472998    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 22:22:57.473140 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.473138    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 22:22:57.473426 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.473349    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wj42\""
Nov 01 22:22:57.473426 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.473359    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 22:22:57.473537 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.473487    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm5kv\""
Nov 01 22:22:57.473608 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.473590    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftrlf\""
Nov 01 22:22:57.473662 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.473596    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w2xg9\""
Nov 01 22:22:57.473662 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.473602    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 22:22:57.475040 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.475025    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-67p2k\""
Nov 01 22:22:57.478872 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.478857    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 22:22:57.479009 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.478994    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 22:22:57.479009 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.479003    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 22:22:57.479373 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.479359    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 22:22:57.479435 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.479391    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hb4ll\""
Nov 01 22:22:57.479435 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.479392    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:22:57.479435 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.479414    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 22:22:57.479435 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.479389    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 22:22:57.515986 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.515963    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 22:22:57.516148 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516107    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 22:22:57.516148 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516115    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 22:22:57.516300 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516158    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jbqbs\""
Nov 01 22:22:57.516300 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516173    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 22:22:57.516300 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516203    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ljglf\""
Nov 01 22:22:57.516300 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516241    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 22:22:57.516445 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516325    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n4xsx\""
Nov 01 22:22:57.516445 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516359    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 22:22:57.516445 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516385    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 22:22:57.516445 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.516399    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 22:22:57.517454 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.517439    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 22:22:57.517508 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.517471    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hm29x"
Nov 01 22:22:57.529483 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqct\" (UniqueName: \"kubernetes.io/projected/bb0df7fd-8362-4c2a-8258-e79901e6c402-kube-api-access-tvqct\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.529541 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x6dp\" (UniqueName: \"kubernetes.io/projected/b0f6996e-a749-41be-9a50-eaa334e1f584-kube-api-access-6x6dp\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.529541 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.529541 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-k8s-cni-cncf-io\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.529687 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-tmp\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.529687 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:57.529687 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-conf-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.529687 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-cnibin\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.529687 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.529687 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-etc-tuned\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.529700    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529729    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-cnibin\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.529775    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:58.029754433 +0000 UTC m=+3.108754385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-kubelet\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-systemd-units\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529848    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-log-socket\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.529932 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529911    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-config\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-system-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-kubelet\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.529989    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/a9038753-40e0-4e1f-aacc-da2a573be477-kube-api-access-mds7f\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-multus\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wjk\" (UniqueName: \"kubernetes.io/projected/244d4caf-546f-46aa-bdf2-8023694c22a1-kube-api-access-m7wjk\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb0df7fd-8362-4c2a-8258-e79901e6c402-hosts-file\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-modprobe-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.530356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-host\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-lib-modules\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-cnibin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-cni-binary-copy\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-hostroot\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-system-cni-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530352    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb0df7fd-8362-4c2a-8258-e79901e6c402-hosts-file\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530589    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-modprobe-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530596    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-system-cni-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530635    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-host\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m46p\" (UniqueName: \"kubernetes.io/projected/28838384-0663-4a33-9903-26afb7288d9e-kube-api-access-6m46p\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ee57462-5ba3-4cfe-a233-13c046a7e797-agent-certs\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530718    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-lib-modules\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530815    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-systemd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.530954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-kubelet-config\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530964    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.530994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-netns\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531021    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-kubelet-config\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-conf\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-ovn\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531146    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-conf\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0df7fd-8362-4c2a-8258-e79901e6c402-tmp-dir\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531164    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ee57462-5ba3-4cfe-a233-13c046a7e797-konnectivity-ca\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-node-log\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-netd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531264    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-env-overrides\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-etc-kubernetes\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/244d4caf-546f-46aa-bdf2-8023694c22a1-iptables-alerter-script\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.531832 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-run\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531437    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0df7fd-8362-4c2a-8258-e79901e6c402-tmp-dir\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-device-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531492    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-device-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tds8r\" (UniqueName: \"kubernetes.io/projected/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-kube-api-access-tds8r\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-netns\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-socket-dir-parent\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysconfig\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-host\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-etc-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-bin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwbz\" (UniqueName: \"kubernetes.io/projected/93e7ee65-e872-4029-b310-a6f04d32c8fd-kube-api-access-ttwbz\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531739    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ee57462-5ba3-4cfe-a233-13c046a7e797-konnectivity-ca\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-systemd\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-var-lib-kubelet\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.532679 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6jq\" (UniqueName: \"kubernetes.io/projected/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kube-api-access-nn6jq\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531965    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/244d4caf-546f-46aa-bdf2-8023694c22a1-iptables-alerter-script\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531976    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-run\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531987    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.531801    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-systemd\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532151    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysconfig\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532201    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.532201    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532239    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532254    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.532274    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret podName:c3e0e60c-b299-48da-86b6-3cf0dfdfa238 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:58.032256926 +0000 UTC m=+3.111256879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret") pod "global-pull-secret-syncer-njrpj" (UID: "c3e0e60c-b299-48da-86b6-3cf0dfdfa238") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532291    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-var-lib-kubelet\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532303    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-host\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-slash\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.533292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/244d4caf-546f-46aa-bdf2-8023694c22a1-host-slash\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-sys\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532477    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532481    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/244d4caf-546f-46aa-bdf2-8023694c22a1-host-slash\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-serviceca\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532527    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-sys\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-dbus\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfc7\" (UniqueName: \"kubernetes.io/projected/990ac23b-9dab-47c9-a8bf-13907483a733-kube-api-access-nhfc7\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-bin\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0f6996e-a749-41be-9a50-eaa334e1f584-ovn-node-metrics-cert\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-kubernetes\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532655    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-dbus\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-var-lib-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-script-lib\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-daemon-config\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532771    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-kubernetes\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.533783 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-multus-certs\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.534298 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.532956    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-serviceca\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.534298 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.533850    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-tmp\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.534298 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.533868    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-etc-tuned\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.534298 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.534013    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ee57462-5ba3-4cfe-a233-13c046a7e797-agent-certs\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:22:57.592010 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.591958    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:57.592010 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.591979    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:57.592010 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.591989    2583 projected.go:196] Error preparing data for projected volume kube-api-access-hxp7p for pod openshift-network-diagnostics/network-check-target-j8gn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:57.592231 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:57.592034    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p podName:6854396e-1bc7-4bab-8cd7-68d758d7d74f nodeName:}" failed. No retries permitted until 2026-11-01 22:22:58.092017202 +0000 UTC m=+3.171017171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hxp7p" (UniqueName: "kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p") pod "network-check-target-j8gn5" (UID: "6854396e-1bc7-4bab-8cd7-68d758d7d74f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:57.593858 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.593839    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqct\" (UniqueName: \"kubernetes.io/projected/bb0df7fd-8362-4c2a-8258-e79901e6c402-kube-api-access-tvqct\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.594168 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.594147    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m46p\" (UniqueName: \"kubernetes.io/projected/28838384-0663-4a33-9903-26afb7288d9e-kube-api-access-6m46p\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.594258 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.594214    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/a9038753-40e0-4e1f-aacc-da2a573be477-kube-api-access-mds7f\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.595769 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.595750    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wjk\" (UniqueName: \"kubernetes.io/projected/244d4caf-546f-46aa-bdf2-8023694c22a1-kube-api-access-m7wjk\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.597172 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.597151    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tds8r\" (UniqueName: \"kubernetes.io/projected/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-kube-api-access-tds8r\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.598096 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.598071    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6jq\" (UniqueName: \"kubernetes.io/projected/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kube-api-access-nn6jq\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.598972 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.598956    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfc7\" (UniqueName: \"kubernetes.io/projected/990ac23b-9dab-47c9-a8bf-13907483a733-kube-api-access-nhfc7\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:57.633110 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-ovn\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633195 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-node-log\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633195 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-netd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633195 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633156    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-ovn\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633195 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633175    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-node-log\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633195 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-env-overrides\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633195    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-netd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-etc-kubernetes\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-netns\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-socket-dir-parent\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633266    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-etc-kubernetes\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-etc-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633296    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-netns\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-bin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633331    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-etc-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwbz\" (UniqueName: \"kubernetes.io/projected/93e7ee65-e872-4029-b310-a6f04d32c8fd-kube-api-access-ttwbz\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633350 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633310    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-socket-dir-parent\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-slash\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-bin\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0f6996e-a749-41be-9a50-eaa334e1f584-ovn-node-metrics-cert\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633427    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-slash\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633447    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-bin\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633337    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-bin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-var-lib-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-script-lib\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-daemon-config\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633564    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-var-lib-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633565    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-env-overrides\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-multus-certs\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x6dp\" (UniqueName: \"kubernetes.io/projected/b0f6996e-a749-41be-9a50-eaa334e1f584-kube-api-access-6x6dp\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633637    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-multus-certs\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-k8s-cni-cncf-io\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.633730 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633728    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-conf-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633790    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-k8s-cni-cncf-io\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-kubelet\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633808    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-conf-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-systemd-units\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633850    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-kubelet\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-log-socket\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633888    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-log-socket\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633894    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-systemd-units\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-config\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-system-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.633980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-kubelet\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-multus\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634009    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-system-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634061    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-kubelet\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634067    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-script-lib\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.634529 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-cnibin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634096    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-cni-binary-copy\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634111    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-daemon-config\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634110    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-multus\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634137    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-cnibin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-hostroot\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634150    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-hostroot\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-systemd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-netns\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634322    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-netns\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634373    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-systemd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634389    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-config\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634403    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634430    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.635084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.634602    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-cni-binary-copy\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.635652 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.635306    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0f6996e-a749-41be-9a50-eaa334e1f584-ovn-node-metrics-cert\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.696894 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.696866    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x6dp\" (UniqueName: \"kubernetes.io/projected/b0f6996e-a749-41be-9a50-eaa334e1f584-kube-api-access-6x6dp\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.697035 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.697018    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwbz\" (UniqueName: \"kubernetes.io/projected/93e7ee65-e872-4029-b310-a6f04d32c8fd-kube-api-access-ttwbz\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.718139 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.718123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:22:57.723432 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.723413    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:22:57.726608 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.726584    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc026e341_04cc_4a50_bb3e_1ff663c2c7e9.slice/crio-6dc5e9b7dc6836b44787ec04bf937d6a43d1d180466bda7c0994ac18a61df6d0 WatchSource:0}: Error finding container 6dc5e9b7dc6836b44787ec04bf937d6a43d1d180466bda7c0994ac18a61df6d0: Status 404 returned error can't find the container with id 6dc5e9b7dc6836b44787ec04bf937d6a43d1d180466bda7c0994ac18a61df6d0
Nov 01 22:22:57.730024 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.730004    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:22:57.731611 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.731592    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee57462_5ba3_4cfe_a233_13c046a7e797.slice/crio-8d0f28e49f9a66ba975f5344a1ef326078b77cf1165f1f1d8701baa492a633fb WatchSource:0}: Error finding container 8d0f28e49f9a66ba975f5344a1ef326078b77cf1165f1f1d8701baa492a633fb: Status 404 returned error can't find the container with id 8d0f28e49f9a66ba975f5344a1ef326078b77cf1165f1f1d8701baa492a633fb
Nov 01 22:22:57.734849 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.734831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:22:57.737508 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.737489    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244d4caf_546f_46aa_bdf2_8023694c22a1.slice/crio-6cfdd2ef206e9e1cf12d1d885bc6d38cfbb51e77c76cc977545a1e440c0ecf33 WatchSource:0}: Error finding container 6cfdd2ef206e9e1cf12d1d885bc6d38cfbb51e77c76cc977545a1e440c0ecf33: Status 404 returned error can't find the container with id 6cfdd2ef206e9e1cf12d1d885bc6d38cfbb51e77c76cc977545a1e440c0ecf33
Nov 01 22:22:57.740851 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.740831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:22:57.741843 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.741824    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0df7fd_8362_4c2a_8258_e79901e6c402.slice/crio-fb69500a3ffaad1b5b2997ea7613c5fa7e69c4d874127a8c77fc8060ffa26fa1 WatchSource:0}: Error finding container fb69500a3ffaad1b5b2997ea7613c5fa7e69c4d874127a8c77fc8060ffa26fa1: Status 404 returned error can't find the container with id fb69500a3ffaad1b5b2997ea7613c5fa7e69c4d874127a8c77fc8060ffa26fa1
Nov 01 22:22:57.747983 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.747964    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b5233e_d5ac_4fa6_adff_97b78dcd1e40.slice/crio-6a0790919b718dbf72ffd4c0f5103bc6c863366d715bb1e4d57edd6a3fd1aa07 WatchSource:0}: Error finding container 6a0790919b718dbf72ffd4c0f5103bc6c863366d715bb1e4d57edd6a3fd1aa07: Status 404 returned error can't find the container with id 6a0790919b718dbf72ffd4c0f5103bc6c863366d715bb1e4d57edd6a3fd1aa07
Nov 01 22:22:57.753524 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.753509    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:22:57.755904 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.755887    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:22:57.761472 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.761452    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28838384_0663_4a33_9903_26afb7288d9e.slice/crio-2d353c680b6240a3b8d6d689c0cfc5933b31575176ef6e8f002e32bf6aab4601 WatchSource:0}: Error finding container 2d353c680b6240a3b8d6d689c0cfc5933b31575176ef6e8f002e32bf6aab4601: Status 404 returned error can't find the container with id 2d353c680b6240a3b8d6d689c0cfc5933b31575176ef6e8f002e32bf6aab4601
Nov 01 22:22:57.763404 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.763384    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9038753_40e0_4e1f_aacc_da2a573be477.slice/crio-de3ddd8feca1a2e812682817453a33d53c74ae5bb5347be6adc185b19c097dd1 WatchSource:0}: Error finding container de3ddd8feca1a2e812682817453a33d53c74ae5bb5347be6adc185b19c097dd1: Status 404 returned error can't find the container with id de3ddd8feca1a2e812682817453a33d53c74ae5bb5347be6adc185b19c097dd1
Nov 01 22:22:57.769019 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.769003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r5xnp"
Nov 01 22:22:57.771221 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:57.771202    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:22:57.779143 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.779111    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e7ee65_e872_4029_b310_a6f04d32c8fd.slice/crio-9b9aa06f8440a40b1b1675a89c33fda35b40278dfec730e2843f55df97eff43c WatchSource:0}: Error finding container 9b9aa06f8440a40b1b1675a89c33fda35b40278dfec730e2843f55df97eff43c: Status 404 returned error can't find the container with id 9b9aa06f8440a40b1b1675a89c33fda35b40278dfec730e2843f55df97eff43c
Nov 01 22:22:57.781827 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:22:57.781731    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f6996e_a749_41be_9a50_eaa334e1f584.slice/crio-f669084e6e5b5a29626cbbe2e84e65a261caf34147b8c1e0f8fe4f1fc92fad8d WatchSource:0}: Error finding container f669084e6e5b5a29626cbbe2e84e65a261caf34147b8c1e0f8fe4f1fc92fad8d: Status 404 returned error can't find the container with id f669084e6e5b5a29626cbbe2e84e65a261caf34147b8c1e0f8fe4f1fc92fad8d
Nov 01 22:22:58.037405 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.037376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:58.037556 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.037435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:58.037556 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:58.037521    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:58.037556 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:58.037525    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:58.037673 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:58.037564    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret podName:c3e0e60c-b299-48da-86b6-3cf0dfdfa238 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.037551819 +0000 UTC m=+4.116551769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret") pod "global-pull-secret-syncer-njrpj" (UID: "c3e0e60c-b299-48da-86b6-3cf0dfdfa238") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:58.037673 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:58.037578    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.037571749 +0000 UTC m=+4.116571699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:58.137822 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.137795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:22:58.137963 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:58.137917    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:58.137963 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:58.137932    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:58.137963 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:58.137942    2583 projected.go:196] Error preparing data for projected volume kube-api-access-hxp7p for pod openshift-network-diagnostics/network-check-target-j8gn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:58.138160 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:58.137989    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p podName:6854396e-1bc7-4bab-8cd7-68d758d7d74f nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.13797353 +0000 UTC m=+4.216973481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxp7p" (UniqueName: "kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p") pod "network-check-target-j8gn5" (UID: "6854396e-1bc7-4bab-8cd7-68d758d7d74f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:58.447859 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.447825    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hnmh2" event={"ID":"2ee57462-5ba3-4cfe-a233-13c046a7e797","Type":"ContainerStarted","Data":"8d0f28e49f9a66ba975f5344a1ef326078b77cf1165f1f1d8701baa492a633fb"}
Nov 01 22:22:58.450911 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.450786    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb" event={"ID":"c026e341-04cc-4a50-bb3e-1ff663c2c7e9","Type":"ContainerStarted","Data":"6dc5e9b7dc6836b44787ec04bf937d6a43d1d180466bda7c0994ac18a61df6d0"}
Nov 01 22:22:58.452288 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.452200    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"f669084e6e5b5a29626cbbe2e84e65a261caf34147b8c1e0f8fe4f1fc92fad8d"}
Nov 01 22:22:58.453541 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.453514    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8g2k7" event={"ID":"a9038753-40e0-4e1f-aacc-da2a573be477","Type":"ContainerStarted","Data":"de3ddd8feca1a2e812682817453a33d53c74ae5bb5347be6adc185b19c097dd1"}
Nov 01 22:22:58.454809 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.454780    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-km7mv" event={"ID":"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40","Type":"ContainerStarted","Data":"6a0790919b718dbf72ffd4c0f5103bc6c863366d715bb1e4d57edd6a3fd1aa07"}
Nov 01 22:22:58.456220 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.456187    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tb9n7" event={"ID":"244d4caf-546f-46aa-bdf2-8023694c22a1","Type":"ContainerStarted","Data":"6cfdd2ef206e9e1cf12d1d885bc6d38cfbb51e77c76cc977545a1e440c0ecf33"}
Nov 01 22:22:58.457626 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.457597    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r5xnp" event={"ID":"93e7ee65-e872-4029-b310-a6f04d32c8fd","Type":"ContainerStarted","Data":"9b9aa06f8440a40b1b1675a89c33fda35b40278dfec730e2843f55df97eff43c"}
Nov 01 22:22:58.458991 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.458961    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zd476" event={"ID":"28838384-0663-4a33-9903-26afb7288d9e","Type":"ContainerStarted","Data":"2d353c680b6240a3b8d6d689c0cfc5933b31575176ef6e8f002e32bf6aab4601"}
Nov 01 22:22:58.460261 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.460235    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-njpz8" event={"ID":"bb0df7fd-8362-4c2a-8258-e79901e6c402","Type":"ContainerStarted","Data":"fb69500a3ffaad1b5b2997ea7613c5fa7e69c4d874127a8c77fc8060ffa26fa1"}
Nov 01 22:22:58.519933 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.519884    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:58 +0000 UTC" deadline="2028-07-21 04:01:30.146673458 +0000 UTC"
Nov 01 22:22:58.519933 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:58.519933    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15053h38m31.626745639s"
Nov 01 22:22:59.045150 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:59.045111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:59.045345 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.045270    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:59.045414 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.045360    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:01.045338585 +0000 UTC m=+6.124338540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:59.045486 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:59.045436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:59.045540 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.045522    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:59.045586 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.045561    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret podName:c3e0e60c-b299-48da-86b6-3cf0dfdfa238 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:01.045548754 +0000 UTC m=+6.124548708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret") pod "global-pull-secret-syncer-njrpj" (UID: "c3e0e60c-b299-48da-86b6-3cf0dfdfa238") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:59.146709 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:59.146674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:22:59.146878 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.146829    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:59.146878 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.146847    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:59.146878 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.146858    2583 projected.go:196] Error preparing data for projected volume kube-api-access-hxp7p for pod openshift-network-diagnostics/network-check-target-j8gn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:59.147059 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.146908    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p podName:6854396e-1bc7-4bab-8cd7-68d758d7d74f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:01.146894183 +0000 UTC m=+6.225894139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxp7p" (UniqueName: "kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p") pod "network-check-target-j8gn5" (UID: "6854396e-1bc7-4bab-8cd7-68d758d7d74f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:59.440482 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:59.440282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:22:59.440482 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:59.440322    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:22:59.440482 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:59.440283    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:22:59.440482 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.440430    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:22:59.440892 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.440526    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:22:59.440892 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:22:59.440624    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:22:59.520941 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:59.520894    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:58 +0000 UTC" deadline="2028-07-25 04:21:09.953686916 +0000 UTC"
Nov 01 22:22:59.520941 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:22:59.520939    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15149h58m10.432753547s"
Nov 01 22:23:01.061248 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:01.060732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:01.061248 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:01.060798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:01.061248 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.060975    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:01.061248 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.061047    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:05.061028023 +0000 UTC m=+10.140027989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:01.061248 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.061131    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:01.061248 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.061164    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret podName:c3e0e60c-b299-48da-86b6-3cf0dfdfa238 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:05.061156513 +0000 UTC m=+10.140156463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret") pod "global-pull-secret-syncer-njrpj" (UID: "c3e0e60c-b299-48da-86b6-3cf0dfdfa238") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:01.161549 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:01.161233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:01.161549 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.161400    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:01.161549 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.161421    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:01.161549 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.161434    2583 projected.go:196] Error preparing data for projected volume kube-api-access-hxp7p for pod openshift-network-diagnostics/network-check-target-j8gn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:01.161549 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.161492    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p podName:6854396e-1bc7-4bab-8cd7-68d758d7d74f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:05.161473572 +0000 UTC m=+10.240473543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxp7p" (UniqueName: "kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p") pod "network-check-target-j8gn5" (UID: "6854396e-1bc7-4bab-8cd7-68d758d7d74f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:01.442316 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:01.442016    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:01.442462 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.442387    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:01.442543 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:01.442480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:01.442596 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.442556    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:01.442651 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:01.442613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:01.442706 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:01.442694    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:01.485770 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:01.484957    2583 generic.go:356] "Generic (PLEG): container finished" podID="d0a42aef018761dafd58eb8c13dacd2c" containerID="108e6909deb4fcdb3c8015d555de35b0c723f82a802b075c217c97909a221ba4" exitCode=0
Nov 01 22:23:01.485770 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:01.485014    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"d0a42aef018761dafd58eb8c13dacd2c","Type":"ContainerDied","Data":"108e6909deb4fcdb3c8015d555de35b0c723f82a802b075c217c97909a221ba4"}
Nov 01 22:23:02.493326 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:02.492412    2583 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="f9e25283528c4029380e4bffea9a4e63ba39a41fc1617693669e07f76ce1993a" exitCode=0
Nov 01 22:23:02.495483 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:02.493107    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zd476" event={"ID":"28838384-0663-4a33-9903-26afb7288d9e","Type":"ContainerDied","Data":"f9e25283528c4029380e4bffea9a4e63ba39a41fc1617693669e07f76ce1993a"}
Nov 01 22:23:02.499255 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:02.498754    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"0dcabca36fe7386806bacef405dbcfa5","Type":"ContainerStarted","Data":"01a9a90ac18f71932de7cab9c04fbfbf11fcc24473dfa7ed8e062d4c958b0014"}
Nov 01 22:23:02.503795 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:02.503337    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"d0a42aef018761dafd58eb8c13dacd2c","Type":"ContainerStarted","Data":"fe486dfccbff84f7b7244af7945cb79fcb03690a29401ec8b1ceb437b5cd193e"}
Nov 01 22:23:02.935340 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:02.933931    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal" podStartSLOduration=5.933908042 podStartE2EDuration="5.933908042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:02.808476301 +0000 UTC m=+7.887476276" watchObservedRunningTime="2026-11-01 22:23:02.933908042 +0000 UTC m=+8.012908017"
Nov 01 22:23:03.440607 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:03.440152    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:03.440607 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:03.440261    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:03.440607 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:03.440331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:03.440607 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:03.440410    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:03.440607 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:03.440498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:03.440607 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:03.440580    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:03.510984 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:03.510537    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hnmh2" event={"ID":"2ee57462-5ba3-4cfe-a233-13c046a7e797","Type":"ContainerStarted","Data":"ea405ae99a9fc87e4cf07d3fb598767b5578d0d8eba7e7a6b01e1a976723255a"}
Nov 01 22:23:03.643141 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:03.639953    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal" podStartSLOduration=6.639936217 podStartE2EDuration="6.639936217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:02.935321892 +0000 UTC m=+8.014321872" watchObservedRunningTime="2026-11-01 22:23:03.639936217 +0000 UTC m=+8.718936190"
Nov 01 22:23:03.643141 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:03.640718    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hnmh2" podStartSLOduration=2.412080835 podStartE2EDuration="7.640710198s" totalImagesPullingTime="5.228629363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:03.637659733 +0000 UTC m=+8.716659706" watchObservedRunningTime="2026-11-01 22:23:03.640710198 +0000 UTC m=+8.719710175"
Nov 01 22:23:04.515986 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:04.515869    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-km7mv" event={"ID":"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40","Type":"ContainerStarted","Data":"9d9db735e5ff952697dffff855370f51e3a213ac3fa495d5df94dfa2039c443b"}
Nov 01 22:23:05.091534 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:05.091446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:05.091534 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:05.091514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:05.091769 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.091668    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:05.091769 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.091731    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:13.091711506 +0000 UTC m=+18.170711472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:05.091978 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.091903    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:05.091978 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.091951    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret podName:c3e0e60c-b299-48da-86b6-3cf0dfdfa238 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:13.091938463 +0000 UTC m=+18.170938429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret") pod "global-pull-secret-syncer-njrpj" (UID: "c3e0e60c-b299-48da-86b6-3cf0dfdfa238") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:05.192465 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:05.192395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:05.192639 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.192573    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:05.192639 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.192599    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:05.192639 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.192616    2583 projected.go:196] Error preparing data for projected volume kube-api-access-hxp7p for pod openshift-network-diagnostics/network-check-target-j8gn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:05.192821 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.192679    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p podName:6854396e-1bc7-4bab-8cd7-68d758d7d74f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:13.192661278 +0000 UTC m=+18.271661281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxp7p" (UniqueName: "kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p") pod "network-check-target-j8gn5" (UID: "6854396e-1bc7-4bab-8cd7-68d758d7d74f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:05.444331 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:05.442635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:05.444331 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.442774    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:05.444331 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:05.442865    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:05.444331 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.442954    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:05.444331 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:05.443014    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:05.444331 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:05.443101    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:05.521567 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:05.521442    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb" event={"ID":"c026e341-04cc-4a50-bb3e-1ff663c2c7e9","Type":"ContainerStarted","Data":"592b17023c8a77031d7ce36370e0d67e9d72ac3a824b8f758860554823e1ae75"}
Nov 01 22:23:05.853348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:05.852681    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:06.293051 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:06.292686    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:06.293287 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:06.293173    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:06.418321 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:06.417523    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-km7mv" podStartSLOduration=4.232819249 podStartE2EDuration="10.417502227s" totalImagesPullingTime="6.184682978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:04.643954532 +0000 UTC m=+9.722954507" watchObservedRunningTime="2026-11-01 22:23:06.417502227 +0000 UTC m=+11.496502202"
Nov 01 22:23:06.539147 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:06.529766    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r5xnp" event={"ID":"93e7ee65-e872-4029-b310-a6f04d32c8fd","Type":"ContainerStarted","Data":"bc517ec35b24e5601df9cda7ce0c52fbd40becc59645287d337dd9a7d2d72b54"}
Nov 01 22:23:06.539147 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:06.532956    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:06.666497 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:06.666433    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-r5xnp" podStartSLOduration=1.031796911 podStartE2EDuration="9.666414076s" totalImagesPullingTime="8.634617165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:06.658808995 +0000 UTC m=+11.737808969" watchObservedRunningTime="2026-11-01 22:23:06.666414076 +0000 UTC m=+11.745414050"
Nov 01 22:23:07.179193 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.179166    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 22:23:07.378553 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.378389    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:07.179188037Z","UUID":"68ed6d30-7085-4f3e-b77a-b63d77488227","Handler":null,"Name":"","Endpoint":""}
Nov 01 22:23:07.380151 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.380127    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 22:23:07.380269 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.380186    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 22:23:07.440296 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.440267    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:07.440436 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.440338    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:07.440436 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.440347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:07.440564 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:07.440457    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:07.440623 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:07.440575    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:07.440717 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:07.440682    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:07.534225 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.534196    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-njpz8" event={"ID":"bb0df7fd-8362-4c2a-8258-e79901e6c402","Type":"ContainerStarted","Data":"9ba6b1e436efbd4521fc84cb1aebdfb2d581d8233cad742a169d2f5afc71a446"}
Nov 01 22:23:07.537195 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.536732    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb" event={"ID":"c026e341-04cc-4a50-bb3e-1ff663c2c7e9","Type":"ContainerStarted","Data":"0b48ddafa966b94f699a5c1cadc77751dc7a85a61bbb53bd790ff26bde93bd3b"}
Nov 01 22:23:07.658925 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:07.658806    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-njpz8" podStartSLOduration=2.576174461 podStartE2EDuration="11.658787582s" totalImagesPullingTime="9.082613121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:07.65851407 +0000 UTC m=+12.737514046" watchObservedRunningTime="2026-11-01 22:23:07.658787582 +0000 UTC m=+12.737787557"
Nov 01 22:23:08.541719 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:08.541481    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tb9n7" event={"ID":"244d4caf-546f-46aa-bdf2-8023694c22a1","Type":"ContainerStarted","Data":"1d5541094f023686e8801fc0586a8f4731d14702b1fb3b20827d2f880a55ef92"}
Nov 01 22:23:08.544596 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:08.544568    2583 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="e07f0ffd5ce5ee0638ca44fe20586c14ebb26669881e9c96a6e4b835adfa2d8f" exitCode=0
Nov 01 22:23:08.544748 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:08.544662    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zd476" event={"ID":"28838384-0663-4a33-9903-26afb7288d9e","Type":"ContainerDied","Data":"e07f0ffd5ce5ee0638ca44fe20586c14ebb26669881e9c96a6e4b835adfa2d8f"}
Nov 01 22:23:08.780856 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:08.780794    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tb9n7" podStartSLOduration=3.4522423079999998 podStartE2EDuration="12.780776122s" totalImagesPullingTime="9.328533814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:08.657635422 +0000 UTC m=+13.736635394" watchObservedRunningTime="2026-11-01 22:23:08.780776122 +0000 UTC m=+13.859776085"
Nov 01 22:23:09.440257 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:09.440234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:09.440344 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:09.440255    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:09.440377 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:09.440343    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:09.440415 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:09.440391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:09.440494 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:09.440478    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:09.440567 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:09.440551    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:09.549094 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:09.549058    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb" event={"ID":"c026e341-04cc-4a50-bb3e-1ff663c2c7e9","Type":"ContainerStarted","Data":"96353285072d2b5c0d0f9f02894797820ff21d3afe1ccdf269ac13b368921110"}
Nov 01 22:23:09.550364 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:09.550337    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8g2k7" event={"ID":"a9038753-40e0-4e1f-aacc-da2a573be477","Type":"ContainerStarted","Data":"7add5be36a0dbdfd7439cabf160fe04770f2611123f7cadea70e910356533d60"}
Nov 01 22:23:09.787345 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:09.787303    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8g2k7" podStartSLOduration=1.177832273 podStartE2EDuration="12.787289932s" totalImagesPullingTime="11.609457659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:09.78688393 +0000 UTC m=+14.865883899" watchObservedRunningTime="2026-11-01 22:23:09.787289932 +0000 UTC m=+14.866289902"
Nov 01 22:23:09.787842 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:09.787677    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb" podStartSLOduration=2.507920645 podStartE2EDuration="13.787668164s" totalImagesPullingTime="11.279747519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:09.66769416 +0000 UTC m=+14.746694130" watchObservedRunningTime="2026-11-01 22:23:09.787668164 +0000 UTC m=+14.866668137"
Nov 01 22:23:10.554369 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:10.554341    2583 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="b16765d00e587e02554ee3c027da4d0c15378971cf750453546e511116c392a2" exitCode=0
Nov 01 22:23:10.554486 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:10.554398    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zd476" event={"ID":"28838384-0663-4a33-9903-26afb7288d9e","Type":"ContainerDied","Data":"b16765d00e587e02554ee3c027da4d0c15378971cf750453546e511116c392a2"}
Nov 01 22:23:11.439764 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:11.439585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:11.439764 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:11.439597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:11.439764 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:11.439718    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:11.439764 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:11.439737    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:11.440611 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:11.439837    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:11.440611 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:11.439909    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:12.561784 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:12.561584    2583 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="2ad613449ebaa2a8e08c12e4d8438594c04f2e022f1cbd29cbb8dfc5c5b33dde" exitCode=0
Nov 01 22:23:12.561784 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:12.561671    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zd476" event={"ID":"28838384-0663-4a33-9903-26afb7288d9e","Type":"ContainerDied","Data":"2ad613449ebaa2a8e08c12e4d8438594c04f2e022f1cbd29cbb8dfc5c5b33dde"}
Nov 01 22:23:13.161842 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:13.161811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:13.162029 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:13.161861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:13.162029 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.161962    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:13.162029 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.161976    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:13.162029 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.162024    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret podName:c3e0e60c-b299-48da-86b6-3cf0dfdfa238 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:29.162008677 +0000 UTC m=+34.241008631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret") pod "global-pull-secret-syncer-njrpj" (UID: "c3e0e60c-b299-48da-86b6-3cf0dfdfa238") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:13.162257 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.162055    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:29.162033337 +0000 UTC m=+34.241033286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:13.262424 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:13.262395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:13.262569 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.262531    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:13.262569 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.262550    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:13.262569 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.262569    2583 projected.go:196] Error preparing data for projected volume kube-api-access-hxp7p for pod openshift-network-diagnostics/network-check-target-j8gn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:13.262693 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.262613    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p podName:6854396e-1bc7-4bab-8cd7-68d758d7d74f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:29.262598679 +0000 UTC m=+34.341598640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxp7p" (UniqueName: "kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p") pod "network-check-target-j8gn5" (UID: "6854396e-1bc7-4bab-8cd7-68d758d7d74f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:13.439356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:13.439328    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:13.439356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:13.439351    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:13.439566 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:13.439334    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:13.439566 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.439466    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:13.439663 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.439606    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:13.439756 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:13.439720    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:14.569685 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:14.569595    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"7fbe65248436db7538b91c63220b50737916db144a85ac312e215a05acdb4084"}
Nov 01 22:23:14.570142 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:14.569708    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"33cc25b6585473af608ba7cba00c8d815fd5332f67a8d88d0fd765b469753dbd"}
Nov 01 22:23:14.570142 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:14.569753    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"efee397964ead35d492e2053aa788ec0cbd130319ecbf7305aef38f6fbb58383"}
Nov 01 22:23:14.570142 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:14.569800    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"13c11318e677adc2319f3b010be3b82f4366388708437e69f09dfd65be93dc25"}
Nov 01 22:23:14.570142 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:14.569851    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"ea290e118268a9b6e4f2eae9a35489c4dbd04b514f0ab6dc1681b36f5d386d14"}
Nov 01 22:23:14.570142 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:14.569895    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"6cc516f9785931f91940a04524828d7f41fa909a42f00d9532446a06faf5baec"}
Nov 01 22:23:15.440360 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:15.440333    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:15.440525 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:15.440407    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:15.440525 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:15.440498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:15.440606 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:15.440588    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:15.440642 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:15.440635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:15.440702 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:15.440689    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:15.573891 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:15.573856    2583 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="dc8690c2b08fd66e82ecde16cf9647e9cc54173ea4649e39202e8be9a28b86de" exitCode=0
Nov 01 22:23:15.574274 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:15.573901    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zd476" event={"ID":"28838384-0663-4a33-9903-26afb7288d9e","Type":"ContainerDied","Data":"dc8690c2b08fd66e82ecde16cf9647e9cc54173ea4649e39202e8be9a28b86de"}
Nov 01 22:23:16.579474 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:16.579393    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"96edab82796d3fbd1d7120edd524c11b2f136e35de8877bb5c21a67064e62111"}
Nov 01 22:23:16.582162 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:16.582140    2583 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="37dd6859334ce20969d23bdcc2f15e31f3be231c8cc97bf2b5821f0385c73851" exitCode=0
Nov 01 22:23:16.582273 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:16.582177    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zd476" event={"ID":"28838384-0663-4a33-9903-26afb7288d9e","Type":"ContainerDied","Data":"37dd6859334ce20969d23bdcc2f15e31f3be231c8cc97bf2b5821f0385c73851"}
Nov 01 22:23:17.440314 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:17.440279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:17.440314 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:17.440312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:17.440507 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:17.440372    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:17.440507 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:17.440432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:17.440564 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:17.440510    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:17.440611 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:17.440595    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:17.586800 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:17.586765    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zd476" event={"ID":"28838384-0663-4a33-9903-26afb7288d9e","Type":"ContainerStarted","Data":"fd5bedfbcbcec42a7ee58a8c697292ce3d1bb2ea25000d1cd5e02babe394e04f"}
Nov 01 22:23:17.732101 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:17.732009    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zd476" podStartSLOduration=6.341999802 podStartE2EDuration="20.731995719s" totalImagesPullingTime="14.389995917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:17.731543184 +0000 UTC m=+22.810543153" watchObservedRunningTime="2026-11-01 22:23:17.731995719 +0000 UTC m=+22.810995688"
Nov 01 22:23:18.592612 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:18.592577    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" event={"ID":"b0f6996e-a749-41be-9a50-eaa334e1f584","Type":"ContainerStarted","Data":"a4e169017475c3e0fb5b08ff0bc73ba736baeefdceffb01c74e88302ecd9c6f8"}
Nov 01 22:23:18.863728 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:18.863682    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj" podStartSLOduration=5.715497384 podStartE2EDuration="21.863667296s" totalImagesPullingTime="16.148169912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:18.811982704 +0000 UTC m=+23.890982675" watchObservedRunningTime="2026-11-01 22:23:18.863667296 +0000 UTC m=+23.942667266"
Nov 01 22:23:18.953819 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:18.953786    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:19.440100 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:19.440071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:19.440352 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:19.440102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:19.440352 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:19.440180    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:19.440352 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:19.440224    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:19.440352 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:19.440291    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:19.440541 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:19.440361    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:19.596233 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:19.596197    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:19.596696 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:19.596282    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:19.612726 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:19.612700    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:19.613280 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:19.613201    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:21.112020 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:21.111994    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-njrpj"]
Nov 01 22:23:21.112397 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:21.112100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:21.112397 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:21.112180    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:21.114143 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:21.114119    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xvhpg"]
Nov 01 22:23:21.114260 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:21.114201    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:21.114318 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:21.114301    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:21.114722 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:21.114704    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j8gn5"]
Nov 01 22:23:21.114804 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:21.114791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:21.114897 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:21.114882    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:22.439950 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:22.439775    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:22.440467 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:22.439799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:22.440467 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:22.440058    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:22.440467 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:22.439833    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:22.440467 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:22.440140    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:22.440467 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:22.440170    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:24.440029 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:24.439998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:24.440482 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:24.439998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:24.440482 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:24.439998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:24.440482 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:24.440126    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njrpj" podUID="c3e0e60c-b299-48da-86b6-3cf0dfdfa238"
Nov 01 22:23:24.440482 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:24.440254    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvhpg" podUID="990ac23b-9dab-47c9-a8bf-13907483a733"
Nov 01 22:23:24.440482 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:24.440346    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j8gn5" podUID="6854396e-1bc7-4bab-8cd7-68d758d7d74f"
Nov 01 22:23:26.218153 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.218128    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeReady"
Nov 01 22:23:26.218488 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.218271    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 22:23:26.333332 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.333258    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ktgdl"]
Nov 01 22:23:26.337650 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.337629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ktgdl"
Nov 01 22:23:26.375847 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.375817    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rhtp4"]
Nov 01 22:23:26.379710 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.379695    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ths57"]
Nov 01 22:23:26.379838 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.379822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rhtp4"
Nov 01 22:23:26.383349 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.383329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:26.385120 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.385102    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d5889b767-99spr"]
Nov 01 22:23:26.388580 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.388563    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwk8z"]
Nov 01 22:23:26.388691 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.388675    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.392344 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.392325    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"]
Nov 01 22:23:26.392457 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.392438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gwk8z"
Nov 01 22:23:26.395664 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.395648    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6n7x4"]
Nov 01 22:23:26.395751 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.395739    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.399036 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.399019    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"]
Nov 01 22:23:26.399137 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.399119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.402208 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.402187    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"]
Nov 01 22:23:26.402327 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.402306    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"
Nov 01 22:23:26.405244 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.405226    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-677954f946-rrb8k"]
Nov 01 22:23:26.405357 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.405331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.408335 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.408318    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.429516 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.429487    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"]
Nov 01 22:23:26.436727 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.435670    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"]
Nov 01 22:23:26.436727 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.436577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.437556 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.437534    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.437669 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.437553    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.440887 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.439508    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56cdb546f-44l9r"]
Nov 01 22:23:26.440887 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.439666    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.442061 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.442032    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 22:23:26.442197 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.442098    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2sjxk\""
Nov 01 22:23:26.443351 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.443333    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 22:23:26.443732 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.443710    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bn2bj\""
Nov 01 22:23:26.443942 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.443843    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 22:23:26.443942 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.443912    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 22:23:26.445199 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.445182    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"]
Nov 01 22:23:26.445336 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.445320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.448514 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.448495    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"]
Nov 01 22:23:26.448941 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.448922    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"
Nov 01 22:23:26.451650 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.451633    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-85ctr"]
Nov 01 22:23:26.451742 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.451730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.454570 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.454552    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ksjkx"]
Nov 01 22:23:26.454668 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.454652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:26.454728 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.454672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:26.454728 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.454718    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:26.454824 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.454779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:26.457795 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.457778    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ktgdl"]
Nov 01 22:23:26.457916 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.457810    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ths57"]
Nov 01 22:23:26.457916 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.457832    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwk8z"]
Nov 01 22:23:26.457916 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.457852    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"]
Nov 01 22:23:26.457916 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.457873    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"]
Nov 01 22:23:26.457916 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.457893    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6n7x4"]
Nov 01 22:23:26.457916 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.457900    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.463847 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.463830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d41ea86a-a166-44b7-a8da-2a7512c9e786-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.463926 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.463856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.463926 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.463875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-snapshots\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.463926 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.463890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.463926 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.463915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.464086 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.464127 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.464159 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.464214 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t44kq\" (UniqueName: \"kubernetes.io/projected/d41ea86a-a166-44b7-a8da-2a7512c9e786-kube-api-access-t44kq\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.464246 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-tmp\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.464296 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.464334 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9679225-9a70-433b-876a-6bd268ae64e4-serving-cert\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.464334 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xmv7\" (UniqueName: \"kubernetes.io/projected/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2-kube-api-access-9xmv7\") pod \"volume-data-source-validator-58c9f5f46d-cwjn4\" (UID: \"cf0a7850-9ba3-4124-b0fe-a088bdfd05f2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"
Nov 01 22:23:26.464410 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.464410 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rnkz\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.464410 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl9z9\" (UniqueName: \"kubernetes.io/projected/c9679225-9a70-433b-876a-6bd268ae64e4-kube-api-access-wl9z9\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.464536 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.464536 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.464627 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7rv\" (UniqueName: \"kubernetes.io/projected/ce89b463-4035-4c17-adc5-4a3e1c07e30b-kube-api-access-cv7rv\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.464627 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:26.464721 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.464721 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8kl5\" (UniqueName: \"kubernetes.io/projected/d66bf75c-5104-4918-b1f5-48fc89bf7f73-kube-api-access-c8kl5\") pod \"kas-connection-checker-685895c7c9-ktgdl\" (UID: \"d66bf75c-5104-4918-b1f5-48fc89bf7f73\") " pod="kube-system/kas-connection-checker-685895c7c9-ktgdl"
Nov 01 22:23:26.464796 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42kc\" (UniqueName: \"kubernetes.io/projected/dc731cae-588c-4d82-850e-1c3c3f954833-kube-api-access-l42kc\") pod \"kas-connection-checker-685895c7c9-rhtp4\" (UID: \"dc731cae-588c-4d82-850e-1c3c3f954833\") " pod="kube-system/kas-connection-checker-685895c7c9-rhtp4"
Nov 01 22:23:26.464870 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:26.464907 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjmv5\" (UniqueName: \"kubernetes.io/projected/1878f36c-168f-490a-bfef-bbe0f477bf88-kube-api-access-qjmv5\") pod \"kas-connection-checker-685895c7c9-gwk8z\" (UID: \"1878f36c-168f-490a-bfef-bbe0f477bf88\") " pod="kube-system/kas-connection-checker-685895c7c9-gwk8z"
Nov 01 22:23:26.464987 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.464912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-service-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.479670 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.479647    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"]
Nov 01 22:23:26.480400 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.480382    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-677954f946-rrb8k"]
Nov 01 22:23:26.487571 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.487295    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"]
Nov 01 22:23:26.489869 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.489560    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rhtp4"]
Nov 01 22:23:26.490333 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.490313    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"]
Nov 01 22:23:26.490888 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.490874    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ksjkx"]
Nov 01 22:23:26.494123 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494079    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.494348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494192    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.494348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494227    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 22:23:26.494348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494259    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.494348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494272    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.494348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494310    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.494348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494332    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494354    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494196    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494371    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494413    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494453    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494416    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494494    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494530    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.494794 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.494556    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.496029 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.495179    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.496029 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.495515    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d5889b767-99spr"]
Nov 01 22:23:26.546312 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.546233    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 22:23:26.566141 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566117    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnp77\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.566251 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.566251 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ced8467-be14-4a8a-a3ae-15566f579cc2-serving-cert\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.566251 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:26.566356 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnkz\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.566387 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.566425 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9679225-9a70-433b-876a-6bd268ae64e4-serving-cert\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.566425 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmv7\" (UniqueName: \"kubernetes.io/projected/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2-kube-api-access-9xmv7\") pod \"volume-data-source-validator-58c9f5f46d-cwjn4\" (UID: \"cf0a7850-9ba3-4124-b0fe-a088bdfd05f2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"
Nov 01 22:23:26.566425 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42kc\" (UniqueName: \"kubernetes.io/projected/dc731cae-588c-4d82-850e-1c3c3f954833-kube-api-access-l42kc\") pod \"kas-connection-checker-685895c7c9-rhtp4\" (UID: \"dc731cae-588c-4d82-850e-1c3c3f954833\") " pod="kube-system/kas-connection-checker-685895c7c9-rhtp4"
Nov 01 22:23:26.566553 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxd8\" (UniqueName: \"kubernetes.io/projected/5b6492e2-6280-45e6-80df-537fa2489fdf-kube-api-access-xtxd8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.566598 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-stats-auth\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.566748 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9z9\" (UniqueName: \"kubernetes.io/projected/c9679225-9a70-433b-876a-6bd268ae64e4-kube-api-access-wl9z9\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.566748 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566743    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:26.566900 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7rv\" (UniqueName: \"kubernetes.io/projected/ce89b463-4035-4c17-adc5-4a3e1c07e30b-kube-api-access-cv7rv\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.566900 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rtr\" (UniqueName: \"kubernetes.io/projected/540ce336-6599-4411-bfc4-a475adfc2ecb-kube-api-access-86rtr\") pod \"network-check-source-6f78d59f99-4b4vw\" (UID: \"540ce336-6599-4411-bfc4-a475adfc2ecb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"
Nov 01 22:23:26.566900 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.566900 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.567095 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.567095 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-cert\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:26.567095 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b330b73a-8364-478a-9589-1b24674fb30e-serving-cert\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.567095 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.566992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmv5\" (UniqueName: \"kubernetes.io/projected/1878f36c-168f-490a-bfef-bbe0f477bf88-kube-api-access-qjmv5\") pod \"kas-connection-checker-685895c7c9-gwk8z\" (UID: \"1878f36c-168f-490a-bfef-bbe0f477bf88\") " pod="kube-system/kas-connection-checker-685895c7c9-gwk8z"
Nov 01 22:23:26.567095 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-service-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.567292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d41ea86a-a166-44b7-a8da-2a7512c9e786-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.567292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-snapshots\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.567292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6492e2-6280-45e6-80df-537fa2489fdf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.567292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2mb\" (UniqueName: \"kubernetes.io/projected/9ced8467-be14-4a8a-a3ae-15566f579cc2-kube-api-access-vq2mb\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.567292 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.567472 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.567472 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.567472 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.567472 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r44ms\" (UniqueName: \"kubernetes.io/projected/2f63221e-2909-4490-bb43-cf2a6164d44c-kube-api-access-r44ms\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.567608 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.567608 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.567496    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:26.567608 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.567511    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:26.567608 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.567608 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t44kq\" (UniqueName: \"kubernetes.io/projected/d41ea86a-a166-44b7-a8da-2a7512c9e786-kube-api-access-t44kq\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.567608 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.567568    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.067551907 +0000 UTC m=+32.146551856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:26.567608 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.567880 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-trusted-ca\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.567880 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-tmp\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.567880 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.567880 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567681    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-snapshots\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.567880 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.567880 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567596    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-service-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.567880 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567798    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d41ea86a-a166-44b7-a8da-2a7512c9e786-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.567880 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgbc\" (UniqueName: \"kubernetes.io/projected/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-kube-api-access-9sgbc\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:26.568151 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.568151 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kl5\" (UniqueName: \"kubernetes.io/projected/d66bf75c-5104-4918-b1f5-48fc89bf7f73-kube-api-access-c8kl5\") pod \"kas-connection-checker-685895c7c9-ktgdl\" (UID: \"d66bf75c-5104-4918-b1f5-48fc89bf7f73\") " pod="kube-system/kas-connection-checker-685895c7c9-ktgdl"
Nov 01 22:23:26.568151 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.567994    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-tmp\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.568151 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b6492e2-6280-45e6-80df-537fa2489fdf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.568151 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ced8467-be14-4a8a-a3ae-15566f579cc2-config\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.568310 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.568310 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568204    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-config\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.568377 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568306    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f63221e-2909-4490-bb43-cf2a6164d44c-tmp-dir\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.568377 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568335    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.568377 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568356    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.568377 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:26.568492 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.568434    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:26.568492 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.568492 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568468    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.568492 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.568490    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.068475895 +0000 UTC m=+32.147475843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:26.568621 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f63221e-2909-4490-bb43-cf2a6164d44c-config-volume\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.568621 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.568621 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.568621 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmbj\" (UniqueName: \"kubernetes.io/projected/b330b73a-8364-478a-9589-1b24674fb30e-kube-api-access-qpmbj\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.568740 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.568740 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.568740 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-default-certificate\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.568740 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.568678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zn5\" (UniqueName: \"kubernetes.io/projected/234fcee4-4ad0-4144-8b1b-7f16e714a08f-kube-api-access-65zn5\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.570508 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.570486    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.570790 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.570775    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.582390 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.582239    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 22:23:26.582390 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.582250    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 22:23:26.582517 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.582360    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 22:23:26.583911 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.583878    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"]
Nov 01 22:23:26.584457 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.584442    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"]
Nov 01 22:23:26.585015 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.584998    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56cdb546f-44l9r"]
Nov 01 22:23:26.588237 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.588221    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.589199 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.589117    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 22:23:26.589199 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.589155    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 22:23:26.589199 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.589192    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 22:23:26.589199 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.589201    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 22:23:26.589378 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.589287    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8dshl\""
Nov 01 22:23:26.589378 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.589320    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.591945 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.591861    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jl5pd\""
Nov 01 22:23:26.591945 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.591866    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j6ll7\""
Nov 01 22:23:26.591945 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.591883    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 22:23:26.591945 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.591908    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmcrl\""
Nov 01 22:23:26.591945 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.591918    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb5v5\""
Nov 01 22:23:26.592140 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.591958    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dhrld\""
Nov 01 22:23:26.592140 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.591980    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tdjb6\""
Nov 01 22:23:26.592564 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592546    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 22:23:26.592663 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592589    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 22:23:26.592762 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592735    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7fqmk\""
Nov 01 22:23:26.592848 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592817    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gqgtr\""
Nov 01 22:23:26.592928 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592879    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 22:23:26.592928 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592902    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 22:23:26.592928 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592907    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hfmjw\""
Nov 01 22:23:26.592928 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592910    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592927    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592949    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592955    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592962    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t4c8s\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.592957    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.593007    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.593019    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.593068    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 22:23:26.593160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.593069    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-25vcq\""
Nov 01 22:23:26.593834 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.593821    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-85ctr"]
Nov 01 22:23:26.597542 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.597527    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 22:23:26.597629 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.597617    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.597699 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.597637    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.597699 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.597644    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.597699 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.597683    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 22:23:26.597699 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.597694    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 22:23:26.597834 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.597729    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.598693 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.598677    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:26.598750 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.598738    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.098719775 +0000 UTC m=+32.177719743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:26.598799 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.598751    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:26.598799 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.598786    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.098775498 +0000 UTC m=+32.177775450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:26.599348 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.599334    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9679225-9a70-433b-876a-6bd268ae64e4-serving-cert\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.626966 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.626941    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmv5\" (UniqueName: \"kubernetes.io/projected/1878f36c-168f-490a-bfef-bbe0f477bf88-kube-api-access-qjmv5\") pod \"kas-connection-checker-685895c7c9-gwk8z\" (UID: \"1878f36c-168f-490a-bfef-bbe0f477bf88\") " pod="kube-system/kas-connection-checker-685895c7c9-gwk8z"
Nov 01 22:23:26.626966 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.626955    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9z9\" (UniqueName: \"kubernetes.io/projected/c9679225-9a70-433b-876a-6bd268ae64e4-kube-api-access-wl9z9\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.627478 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.627456    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.627478 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.627472    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42kc\" (UniqueName: \"kubernetes.io/projected/dc731cae-588c-4d82-850e-1c3c3f954833-kube-api-access-l42kc\") pod \"kas-connection-checker-685895c7c9-rhtp4\" (UID: \"dc731cae-588c-4d82-850e-1c3c3f954833\") " pod="kube-system/kas-connection-checker-685895c7c9-rhtp4"
Nov 01 22:23:26.627605 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.627474    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7rv\" (UniqueName: \"kubernetes.io/projected/ce89b463-4035-4c17-adc5-4a3e1c07e30b-kube-api-access-cv7rv\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:26.627605 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.627575    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmv7\" (UniqueName: \"kubernetes.io/projected/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2-kube-api-access-9xmv7\") pod \"volume-data-source-validator-58c9f5f46d-cwjn4\" (UID: \"cf0a7850-9ba3-4124-b0fe-a088bdfd05f2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"
Nov 01 22:23:26.627723 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.627608    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnkz\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:26.628775 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.628750    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t44kq\" (UniqueName: \"kubernetes.io/projected/d41ea86a-a166-44b7-a8da-2a7512c9e786-kube-api-access-t44kq\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:26.628948 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.628933    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kl5\" (UniqueName: \"kubernetes.io/projected/d66bf75c-5104-4918-b1f5-48fc89bf7f73-kube-api-access-c8kl5\") pod \"kas-connection-checker-685895c7c9-ktgdl\" (UID: \"d66bf75c-5104-4918-b1f5-48fc89bf7f73\") " pod="kube-system/kas-connection-checker-685895c7c9-ktgdl"
Nov 01 22:23:26.650785 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.650767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ktgdl"
Nov 01 22:23:26.669329 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.669407 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmbj\" (UniqueName: \"kubernetes.io/projected/b330b73a-8364-478a-9589-1b24674fb30e-kube-api-access-qpmbj\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.669407 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-default-certificate\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.669502 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zn5\" (UniqueName: \"kubernetes.io/projected/234fcee4-4ad0-4144-8b1b-7f16e714a08f-kube-api-access-65zn5\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.669502 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnp77\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.669599 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.669599 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ced8467-be14-4a8a-a3ae-15566f579cc2-serving-cert\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.669693 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxd8\" (UniqueName: \"kubernetes.io/projected/5b6492e2-6280-45e6-80df-537fa2489fdf-kube-api-access-xtxd8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.669693 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-stats-auth\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.669693 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rtr\" (UniqueName: \"kubernetes.io/projected/540ce336-6599-4411-bfc4-a475adfc2ecb-kube-api-access-86rtr\") pod \"network-check-source-6f78d59f99-4b4vw\" (UID: \"540ce336-6599-4411-bfc4-a475adfc2ecb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"
Nov 01 22:23:26.669880 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.669860    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:26.669939 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.669939 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.669936    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.169915103 +0000 UTC m=+32.248915063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:26.670063 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.669985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-cert\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:26.670063 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b330b73a-8364-478a-9589-1b24674fb30e-serving-cert\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.670163 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6492e2-6280-45e6-80df-537fa2489fdf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.670163 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2mb\" (UniqueName: \"kubernetes.io/projected/9ced8467-be14-4a8a-a3ae-15566f579cc2-kube-api-access-vq2mb\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.670163 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.670163 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.670163 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r44ms\" (UniqueName: \"kubernetes.io/projected/2f63221e-2909-4490-bb43-cf2a6164d44c-kube-api-access-r44ms\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.670163 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-trusted-ca\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgbc\" (UniqueName: \"kubernetes.io/projected/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-kube-api-access-9sgbc\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b6492e2-6280-45e6-80df-537fa2489fdf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ced8467-be14-4a8a-a3ae-15566f579cc2-config\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-config\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f63221e-2909-4490-bb43-cf2a6164d44c-tmp-dir\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.670466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f63221e-2909-4490-bb43-cf2a6164d44c-config-volume\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.671028 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670757    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.671028 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.670993    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f63221e-2909-4490-bb43-cf2a6164d44c-config-volume\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.671711 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.671345    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.171327323 +0000 UTC m=+32.250327294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:26.672230 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.672203    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-trusted-ca\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.672627 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.672606    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b6492e2-6280-45e6-80df-537fa2489fdf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.672802 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.672779    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ced8467-be14-4a8a-a3ae-15566f579cc2-config\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.672872 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.672832    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-config\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.672933 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.672893    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.673001 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.672985    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:26.673076 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.673004    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:26.673076 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.673067    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.173031152 +0000 UTC m=+32.252031114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:26.673190 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.673087    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f63221e-2909-4490-bb43-cf2a6164d44c-tmp-dir\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.673190 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.673144    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:26.673190 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:26.673176    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.173166229 +0000 UTC m=+32.252166177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:26.673417 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.673392    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b330b73a-8364-478a-9589-1b24674fb30e-serving-cert\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.673562 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.673533    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6492e2-6280-45e6-80df-537fa2489fdf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.673851 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.673809    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.673954 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.673927    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ced8467-be14-4a8a-a3ae-15566f579cc2-serving-cert\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.674101 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.674075    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.674196 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.674098    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-default-certificate\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.674663 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.674636    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-stats-auth\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.675125 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.675101    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.675421 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.675401    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-cert\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:26.691556 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.691538    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rhtp4"
Nov 01 22:23:26.709985 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.709965    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gwk8z"
Nov 01 22:23:26.720703 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.720681    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:26.729287 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.729245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"
Nov 01 22:23:26.734121 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.733744    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rtr\" (UniqueName: \"kubernetes.io/projected/540ce336-6599-4411-bfc4-a475adfc2ecb-kube-api-access-86rtr\") pod \"network-check-source-6f78d59f99-4b4vw\" (UID: \"540ce336-6599-4411-bfc4-a475adfc2ecb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"
Nov 01 22:23:26.740644 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.740592    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxd8\" (UniqueName: \"kubernetes.io/projected/5b6492e2-6280-45e6-80df-537fa2489fdf-kube-api-access-xtxd8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.743068 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.742913    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zn5\" (UniqueName: \"kubernetes.io/projected/234fcee4-4ad0-4144-8b1b-7f16e714a08f-kube-api-access-65zn5\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:26.743586 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.743557    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmbj\" (UniqueName: \"kubernetes.io/projected/b330b73a-8364-478a-9589-1b24674fb30e-kube-api-access-qpmbj\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.744352 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.744003    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.744352 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.744293    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r44ms\" (UniqueName: \"kubernetes.io/projected/2f63221e-2909-4490-bb43-cf2a6164d44c-kube-api-access-r44ms\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:26.744493 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.744429    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgbc\" (UniqueName: \"kubernetes.io/projected/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-kube-api-access-9sgbc\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:26.744837 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.744817    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnp77\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:26.751027 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.750972    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2mb\" (UniqueName: \"kubernetes.io/projected/9ced8467-be14-4a8a-a3ae-15566f579cc2-kube-api-access-vq2mb\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.753432 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.753238    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:26.758540 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.758379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:26.772606 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.770928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"
Nov 01 22:23:26.786901 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.777909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:26.786901 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:26.786117    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:27.031479 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:27.031445    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66bf75c_5104_4918_b1f5_48fc89bf7f73.slice/crio-f452eab25f548ccb562744ba964cd762f904182a42a649a2ca173d3363930467 WatchSource:0}: Error finding container f452eab25f548ccb562744ba964cd762f904182a42a649a2ca173d3363930467: Status 404 returned error can't find the container with id f452eab25f548ccb562744ba964cd762f904182a42a649a2ca173d3363930467
Nov 01 22:23:27.034326 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.034300    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ktgdl"]
Nov 01 22:23:27.073119 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.073092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:27.073236 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.073220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:27.073273 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.073233    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:27.073309 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.073302    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.073285411 +0000 UTC m=+33.152285363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:27.073399 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.073358    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:27.073399 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.073375    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:27.073462 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.073428    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.073412488 +0000 UTC m=+33.152412450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:27.092654 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:27.092624    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc731cae_588c_4d82_850e_1c3c3f954833.slice/crio-10509ed8e2ff1cfb0c40c5ac8beb121b55988d2a5e82026c10d42005b226c224 WatchSource:0}: Error finding container 10509ed8e2ff1cfb0c40c5ac8beb121b55988d2a5e82026c10d42005b226c224: Status 404 returned error can't find the container with id 10509ed8e2ff1cfb0c40c5ac8beb121b55988d2a5e82026c10d42005b226c224
Nov 01 22:23:27.094015 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.093992    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rhtp4"]
Nov 01 22:23:27.101551 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.101528    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwk8z"]
Nov 01 22:23:27.109256 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:27.109229    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1878f36c_168f_490a_bfef_bbe0f477bf88.slice/crio-8bb726e0f0e75ba6745e197e0a3b15d44ee248e7453921cb6718acee501f9e49 WatchSource:0}: Error finding container 8bb726e0f0e75ba6745e197e0a3b15d44ee248e7453921cb6718acee501f9e49: Status 404 returned error can't find the container with id 8bb726e0f0e75ba6745e197e0a3b15d44ee248e7453921cb6718acee501f9e49
Nov 01 22:23:27.152695 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.152669    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"]
Nov 01 22:23:27.157246 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.156969    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-6n7x4"]
Nov 01 22:23:27.157609 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.157549    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"]
Nov 01 22:23:27.159510 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:27.159485    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9679225_9a70_433b_876a_6bd268ae64e4.slice/crio-aa90f3d160e5d852429c0d770f93d61fcedb887795c49d8d257fbe55bb1aed94 WatchSource:0}: Error finding container aa90f3d160e5d852429c0d770f93d61fcedb887795c49d8d257fbe55bb1aed94: Status 404 returned error can't find the container with id aa90f3d160e5d852429c0d770f93d61fcedb887795c49d8d257fbe55bb1aed94
Nov 01 22:23:27.164391 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:27.164365    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6492e2_6280_45e6_80df_537fa2489fdf.slice/crio-15e949fd7bd165ff9c0b11d52e4c5441b857da2069d86eb8d9e8b6f3a50ab0df WatchSource:0}: Error finding container 15e949fd7bd165ff9c0b11d52e4c5441b857da2069d86eb8d9e8b6f3a50ab0df: Status 404 returned error can't find the container with id 15e949fd7bd165ff9c0b11d52e4c5441b857da2069d86eb8d9e8b6f3a50ab0df
Nov 01 22:23:27.174402 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.174377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:27.174506 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.174412    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:27.174506 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.174482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:27.174506 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.174501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:27.174669 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.174528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:27.174669 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.174567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:27.174669 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.174666    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:27.174825 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.174718    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.174700088 +0000 UTC m=+33.253700052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:27.174825 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.174786    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:27.174825 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.174822    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.174811144 +0000 UTC m=+33.253811107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:27.174986 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.174872    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:27.174986 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.174896    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.174888422 +0000 UTC m=+33.253888383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:27.174986 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.174958    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.17495177 +0000 UTC m=+33.253951733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:27.175175 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.175000    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:27.175175 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.175030    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.175019565 +0000 UTC m=+33.254019530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:27.175175 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.175092    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:27.175175 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.175099    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:27.175175 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:27.175118    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.175112511 +0000 UTC m=+33.254112473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:27.197655 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:27.197629    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0a7850_9ba3_4124_b0fe_a088bdfd05f2.slice/crio-32b36eebd80dd0f63ac38ffa798818d7e1dbc146597b87182b0479658ae55d21 WatchSource:0}: Error finding container 32b36eebd80dd0f63ac38ffa798818d7e1dbc146597b87182b0479658ae55d21: Status 404 returned error can't find the container with id 32b36eebd80dd0f63ac38ffa798818d7e1dbc146597b87182b0479658ae55d21
Nov 01 22:23:27.200299 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.200244    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"]
Nov 01 22:23:27.211179 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:27.211151    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540ce336_6599_4411_bfc4_a475adfc2ecb.slice/crio-9a38dd3de3f5428a992878a5f1958ee3b87f27e82d339c706feed24fc8ae60c3 WatchSource:0}: Error finding container 9a38dd3de3f5428a992878a5f1958ee3b87f27e82d339c706feed24fc8ae60c3: Status 404 returned error can't find the container with id 9a38dd3de3f5428a992878a5f1958ee3b87f27e82d339c706feed24fc8ae60c3
Nov 01 22:23:27.215084 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.215033    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"]
Nov 01 22:23:27.220204 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.219278    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-85ctr"]
Nov 01 22:23:27.220914 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:27.220845    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ced8467_be14_4a8a_a3ae_15566f579cc2.slice/crio-b6b0092890cfb3ab82ce07038faf4ef3d278826083f772b83dc7b173a4956867 WatchSource:0}: Error finding container b6b0092890cfb3ab82ce07038faf4ef3d278826083f772b83dc7b173a4956867: Status 404 returned error can't find the container with id b6b0092890cfb3ab82ce07038faf4ef3d278826083f772b83dc7b173a4956867
Nov 01 22:23:27.222224 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.221985    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"]
Nov 01 22:23:27.612940 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.612906    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd" event={"ID":"9ced8467-be14-4a8a-a3ae-15566f579cc2","Type":"ContainerStarted","Data":"b6b0092890cfb3ab82ce07038faf4ef3d278826083f772b83dc7b173a4956867"}
Nov 01 22:23:27.614080 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.614053    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw" event={"ID":"540ce336-6599-4411-bfc4-a475adfc2ecb","Type":"ContainerStarted","Data":"9a38dd3de3f5428a992878a5f1958ee3b87f27e82d339c706feed24fc8ae60c3"}
Nov 01 22:23:27.615143 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.615115    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" event={"ID":"b330b73a-8364-478a-9589-1b24674fb30e","Type":"ContainerStarted","Data":"47ba8d729fb8b117d87076fd111776996216e9d31fa46b903b5481ecc3ccfbf2"}
Nov 01 22:23:27.616422 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.616398    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ktgdl" event={"ID":"d66bf75c-5104-4918-b1f5-48fc89bf7f73","Type":"ContainerStarted","Data":"f26537118c033fe1330533c2cb1b95ceb181205d915baa00821e7c37c780dd76"}
Nov 01 22:23:27.616509 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.616439    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ktgdl" event={"ID":"d66bf75c-5104-4918-b1f5-48fc89bf7f73","Type":"ContainerStarted","Data":"f452eab25f548ccb562744ba964cd762f904182a42a649a2ca173d3363930467"}
Nov 01 22:23:27.617522 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.617496    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4" event={"ID":"cf0a7850-9ba3-4124-b0fe-a088bdfd05f2","Type":"ContainerStarted","Data":"32b36eebd80dd0f63ac38ffa798818d7e1dbc146597b87182b0479658ae55d21"}
Nov 01 22:23:27.618594 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.618573    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6n7x4" event={"ID":"c9679225-9a70-433b-876a-6bd268ae64e4","Type":"ContainerStarted","Data":"aa90f3d160e5d852429c0d770f93d61fcedb887795c49d8d257fbe55bb1aed94"}
Nov 01 22:23:27.619732 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.619687    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-85ctr" event={"ID":"cbfec945-9ea9-4a51-a7fd-55714dfa01a6","Type":"ContainerStarted","Data":"e990ee4370f22529169c6c86e23a3b211ac266abef663ecdf16744491f83726d"}
Nov 01 22:23:27.621128 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.621105    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gwk8z" event={"ID":"1878f36c-168f-490a-bfef-bbe0f477bf88","Type":"ContainerStarted","Data":"9778ea22c26d20f4ab234c3ddc8982c06d7be20975d47940b488990bb66a5fdd"}
Nov 01 22:23:27.621223 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.621142    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gwk8z" event={"ID":"1878f36c-168f-490a-bfef-bbe0f477bf88","Type":"ContainerStarted","Data":"8bb726e0f0e75ba6745e197e0a3b15d44ee248e7453921cb6718acee501f9e49"}
Nov 01 22:23:27.622198 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.622174    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb" event={"ID":"5b6492e2-6280-45e6-80df-537fa2489fdf","Type":"ContainerStarted","Data":"15e949fd7bd165ff9c0b11d52e4c5441b857da2069d86eb8d9e8b6f3a50ab0df"}
Nov 01 22:23:27.623565 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.623547    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rhtp4" event={"ID":"dc731cae-588c-4d82-850e-1c3c3f954833","Type":"ContainerStarted","Data":"d59c8769363d7afe3886d3b5639d91d7b7380b50b68fa5605ff33f8ecbad59ec"}
Nov 01 22:23:27.623642 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:27.623581    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rhtp4" event={"ID":"dc731cae-588c-4d82-850e-1c3c3f954833","Type":"ContainerStarted","Data":"10509ed8e2ff1cfb0c40c5ac8beb121b55988d2a5e82026c10d42005b226c224"}
Nov 01 22:23:28.082519 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.082484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:28.082846 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.082582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:28.082846 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.082642    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:28.082846 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.082662    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:28.082846 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.082725    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.082706062 +0000 UTC m=+35.161706027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:28.082846 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.082732    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:28.082846 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.082766    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.082758043 +0000 UTC m=+35.161757996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:28.183856 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.183827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:28.183856 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.183858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:28.184110 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.183884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:28.184110 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.183961    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:28.184110 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.183973    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:28.184110 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.183985    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:28.184110 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184010    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.18399794 +0000 UTC m=+35.262997887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:28.184110 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184058    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.184025063 +0000 UTC m=+35.263025026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:28.184110 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184081    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.184068784 +0000 UTC m=+35.263068734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.184120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.184181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184229    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.184230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184291    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184308    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184318    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.18431047 +0000 UTC m=+35.263310418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184329    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.184324007 +0000 UTC m=+35.263323955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:28.184466 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:28.184339    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.18433449 +0000 UTC m=+35.263334438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:28.574243 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.574197    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rhtp4" podStartSLOduration=426.574182671 podStartE2EDuration="7m6.574182671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:28.573930679 +0000 UTC m=+33.652930649" watchObservedRunningTime="2026-11-01 22:23:28.574182671 +0000 UTC m=+33.653182641"
Nov 01 22:23:28.819541 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:28.817652    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ktgdl" podStartSLOduration=426.817633292 podStartE2EDuration="7m6.817633292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:28.694715738 +0000 UTC m=+33.773715708" watchObservedRunningTime="2026-11-01 22:23:28.817633292 +0000 UTC m=+33.896633263"
Nov 01 22:23:29.194130 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.194100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:29.194303 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.194146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:29.194303 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:29.194270    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:29.194408 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:29.194323    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:24:01.194310142 +0000 UTC m=+66.273310089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : secret "metrics-daemon-secret" not found
Nov 01 22:23:29.197879 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.197852    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:29.213128 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.212921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:29.295891 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.295462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:29.303744 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.303674    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:29.499947 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.499663    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:29.659475 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.658227    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gwk8z" podStartSLOduration=427.658211383 podStartE2EDuration="7m7.658211383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:28.819473303 +0000 UTC m=+33.898473274" watchObservedRunningTime="2026-11-01 22:23:29.658211383 +0000 UTC m=+34.737211353"
Nov 01 22:23:29.659475 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.659407    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-njrpj"]
Nov 01 22:23:29.660154 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:29.660124    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e0e60c_b299_48da_86b6_3cf0dfdfa238.slice/crio-0b59a9fe19dd0a023d76690f99f29fa3a61e7e204f9ea5e0c01832ef4279a428 WatchSource:0}: Error finding container 0b59a9fe19dd0a023d76690f99f29fa3a61e7e204f9ea5e0c01832ef4279a428: Status 404 returned error can't find the container with id 0b59a9fe19dd0a023d76690f99f29fa3a61e7e204f9ea5e0c01832ef4279a428
Nov 01 22:23:29.908514 ip-10-0-129-77 kubenswrapper[2583]: W1101 22:23:29.908478    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6854396e_1bc7_4bab_8cd7_68d758d7d74f.slice/crio-58c85b4f95a81e472be585d174fd402e60520afe34b1d9d1e36738e974253e52 WatchSource:0}: Error finding container 58c85b4f95a81e472be585d174fd402e60520afe34b1d9d1e36738e974253e52: Status 404 returned error can't find the container with id 58c85b4f95a81e472be585d174fd402e60520afe34b1d9d1e36738e974253e52
Nov 01 22:23:29.910011 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:29.909986    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j8gn5"]
Nov 01 22:23:30.103060 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.102988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:30.103235 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.103181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:30.103360 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.103341    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:30.103360 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.103357    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:30.103501 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.103410    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.103392813 +0000 UTC m=+39.182392767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:30.103501 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.103477    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:30.103601 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.103510    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.103499832 +0000 UTC m=+39.182499781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:30.203968 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.203930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:30.204160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.203987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:30.204160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.204054    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:30.204160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.204102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:30.204160 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.204140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:30.204371 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.204172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:30.204429 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204380    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:30.204480 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204445    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.204427226 +0000 UTC m=+39.283427190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:30.204546 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204520    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.204509135 +0000 UTC m=+39.283509083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:30.204602 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204590    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:30.204648 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204624    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.204613843 +0000 UTC m=+39.283613795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:30.204709 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204687    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:30.204709 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204699    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:30.204815 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204727    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.204717189 +0000 UTC m=+39.283717142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:30.204815 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204790    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:30.205000 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204819    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.20480988 +0000 UTC m=+39.283809833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:30.205000 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204873    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:30.205000 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:30.204904    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.204894952 +0000 UTC m=+39.283894900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:30.637681 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.637644    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j8gn5" event={"ID":"6854396e-1bc7-4bab-8cd7-68d758d7d74f","Type":"ContainerStarted","Data":"58c85b4f95a81e472be585d174fd402e60520afe34b1d9d1e36738e974253e52"}
Nov 01 22:23:30.643148 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.643115    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4" event={"ID":"cf0a7850-9ba3-4124-b0fe-a088bdfd05f2","Type":"ContainerStarted","Data":"13034acdd673e6109b1bcad633a78bbc40e197311070c929c40ce6c014fa9ca1"}
Nov 01 22:23:30.644740 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.644709    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-njrpj" event={"ID":"c3e0e60c-b299-48da-86b6-3cf0dfdfa238","Type":"ContainerStarted","Data":"0b59a9fe19dd0a023d76690f99f29fa3a61e7e204f9ea5e0c01832ef4279a428"}
Nov 01 22:23:30.761506 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:30.761414    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4" podStartSLOduration=401.040789993 podStartE2EDuration="6m43.761401126s" totalImagesPullingTime="2.720611133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:30.759859432 +0000 UTC m=+35.838859426" watchObservedRunningTime="2026-11-01 22:23:30.761401126 +0000 UTC m=+35.840401096"
Nov 01 22:23:32.656359 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:32.656329    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/0.log"
Nov 01 22:23:32.656671 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:32.656374    2583 generic.go:356] "Generic (PLEG): container finished" podID="b330b73a-8364-478a-9589-1b24674fb30e" containerID="c79509a55662207969d810e135536d52c33cc32f8be367197a1171cdd1af306b" exitCode=255
Nov 01 22:23:32.656671 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:32.656484    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" event={"ID":"b330b73a-8364-478a-9589-1b24674fb30e","Type":"ContainerDied","Data":"c79509a55662207969d810e135536d52c33cc32f8be367197a1171cdd1af306b"}
Nov 01 22:23:32.656787 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:32.656732    2583 scope.go:122] "RemoveContainer" containerID="c79509a55662207969d810e135536d52c33cc32f8be367197a1171cdd1af306b"
Nov 01 22:23:32.659446 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:32.659290    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6n7x4" event={"ID":"c9679225-9a70-433b-876a-6bd268ae64e4","Type":"ContainerStarted","Data":"cb5b9d48495c813b0abcc7094d7b08715e65df7c6f487c01d4852dc8d4289569"}
Nov 01 22:23:32.898365 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:32.898317    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-6n7x4" podStartSLOduration=400.799199049 podStartE2EDuration="6m45.898300304s" totalImagesPullingTime="5.099101255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:32.89764457 +0000 UTC m=+37.976644539" watchObservedRunningTime="2026-11-01 22:23:32.898300304 +0000 UTC m=+37.977300277"
Nov 01 22:23:33.527232 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.527198    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h84pf"]
Nov 01 22:23:33.549250 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.549227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.605945 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.605900    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h84pf"]
Nov 01 22:23:33.642878 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.642839    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.643003 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.642905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46fb3ebe-f172-40f8-ba72-3a2370cb447a-data-volume\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.643414 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.642983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7r2\" (UniqueName: \"kubernetes.io/projected/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-api-access-rl7r2\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.643544 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.643524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.643623 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.643582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46fb3ebe-f172-40f8-ba72-3a2370cb447a-crio-socket\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.649610 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.649557    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 22:23:33.649610 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.649593    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 22:23:33.655364 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.655341    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwhsh\""
Nov 01 22:23:33.667720 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.667693    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb" event={"ID":"5b6492e2-6280-45e6-80df-537fa2489fdf","Type":"ContainerStarted","Data":"d58508ac467cb86db290f537e090958567b258102af460e71a36060d98b398a9"}
Nov 01 22:23:33.671970 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.671930    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd" event={"ID":"9ced8467-be14-4a8a-a3ae-15566f579cc2","Type":"ContainerStarted","Data":"00c0d4d3492693cc8b67f9fbd8cc146964a8e54f9c170a5ffe9b2caf1585c428"}
Nov 01 22:23:33.674549 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.674528    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/1.log"
Nov 01 22:23:33.675152 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.675123    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/0.log"
Nov 01 22:23:33.675264 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.675191    2583 generic.go:356] "Generic (PLEG): container finished" podID="b330b73a-8364-478a-9589-1b24674fb30e" containerID="d6a6557aeab3afcd4e100f96b6cd67cffa9d11eb2f38f30d74e4cecd52aed739" exitCode=255
Nov 01 22:23:33.675264 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.675235    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" event={"ID":"b330b73a-8364-478a-9589-1b24674fb30e","Type":"ContainerDied","Data":"d6a6557aeab3afcd4e100f96b6cd67cffa9d11eb2f38f30d74e4cecd52aed739"}
Nov 01 22:23:33.675437 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.675297    2583 scope.go:122] "RemoveContainer" containerID="c79509a55662207969d810e135536d52c33cc32f8be367197a1171cdd1af306b"
Nov 01 22:23:33.675773 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.675752    2583 scope.go:122] "RemoveContainer" containerID="d6a6557aeab3afcd4e100f96b6cd67cffa9d11eb2f38f30d74e4cecd52aed739"
Nov 01 22:23:33.675985 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:33.675961    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-m7tdl_openshift-console-operator(b330b73a-8364-478a-9589-1b24674fb30e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" podUID="b330b73a-8364-478a-9589-1b24674fb30e"
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.744993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:33.745081    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:33.745125    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.245112541 +0000 UTC m=+39.324112488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.745317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46fb3ebe-f172-40f8-ba72-3a2370cb447a-data-volume\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.745353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7r2\" (UniqueName: \"kubernetes.io/projected/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-api-access-rl7r2\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.745411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.745473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46fb3ebe-f172-40f8-ba72-3a2370cb447a-crio-socket\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:33.745496    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:33.745526    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.245516889 +0000 UTC m=+39.324516838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.745565    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46fb3ebe-f172-40f8-ba72-3a2370cb447a-crio-socket\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.745749 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.745719    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46fb3ebe-f172-40f8-ba72-3a2370cb447a-data-volume\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.814677 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.814125    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7r2\" (UniqueName: \"kubernetes.io/projected/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-api-access-rl7r2\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:33.914645 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:33.914597    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb" podStartSLOduration=422.043379902 podStartE2EDuration="7m7.914583432s" totalImagesPullingTime="5.87120353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:33.7898687 +0000 UTC m=+38.868868673" watchObservedRunningTime="2026-11-01 22:23:33.914583432 +0000 UTC m=+38.993583401"
Nov 01 22:23:34.031750 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.031697    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd" podStartSLOduration=401.598977489 podStartE2EDuration="6m47.031681344s" totalImagesPullingTime="5.432703855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:34.031274635 +0000 UTC m=+39.110274608" watchObservedRunningTime="2026-11-01 22:23:34.031681344 +0000 UTC m=+39.110681317"
Nov 01 22:23:34.149274 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.149199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:34.149426 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.149323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:34.149426 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.149359    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:34.149426 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.149379    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:34.149426 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.149427    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.149413198 +0000 UTC m=+47.228413150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:34.149623 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.149490    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:34.149623 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.149542    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.149526413 +0000 UTC m=+47.228526378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:34.250414 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.250382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:34.250589 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.250426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:34.250589 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.250460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:34.250589 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.250549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:34.250589 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250554    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:34.250589 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.250589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250604    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250623    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.250603733 +0000 UTC m=+47.329603698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250556    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250642    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.250630983 +0000 UTC m=+47.329630938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.250664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250669    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250695    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:35.250676761 +0000 UTC m=+40.329676718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250710    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250722    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.250712551 +0000 UTC m=+47.329712509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250740    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:35.250730978 +0000 UTC m=+40.329730955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250756    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.250747946 +0000 UTC m=+47.329747941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:34.250807 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.250806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:34.251219 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.250855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:34.251219 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250946    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:34.251219 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.250981    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.250970982 +0000 UTC m=+47.329970937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:34.251219 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.251037    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:34.251219 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.251069    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:34.251219 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.251100    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.251090096 +0000 UTC m=+47.330090045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:34.679957 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.679927    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw" event={"ID":"540ce336-6599-4411-bfc4-a475adfc2ecb","Type":"ContainerStarted","Data":"fcb12148c88a647e66cd0177f5b6329ab0588954100dc3f2e79023b82f69de34"}
Nov 01 22:23:34.681886 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.681863    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/1.log"
Nov 01 22:23:34.682227 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.682198    2583 scope.go:122] "RemoveContainer" containerID="d6a6557aeab3afcd4e100f96b6cd67cffa9d11eb2f38f30d74e4cecd52aed739"
Nov 01 22:23:34.682411 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:34.682390    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-m7tdl_openshift-console-operator(b330b73a-8364-478a-9589-1b24674fb30e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" podUID="b330b73a-8364-478a-9589-1b24674fb30e"
Nov 01 22:23:34.683735 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.683714    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j8gn5" event={"ID":"6854396e-1bc7-4bab-8cd7-68d758d7d74f","Type":"ContainerStarted","Data":"e2b005d0cdf16e74cb7fd77c680fdfe528d41a258f4ec94fa74ee3bc6df8f664"}
Nov 01 22:23:34.683825 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.683791    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:34.685388 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.685368    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-85ctr" event={"ID":"cbfec945-9ea9-4a51-a7fd-55714dfa01a6","Type":"ContainerStarted","Data":"9b3a5fbd832a2195e306c3217cb25d1ca5d54dd655acfec034e7db32fd7f05b4"}
Nov 01 22:23:34.797534 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.797490    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw" podStartSLOduration=398.072011643 podStartE2EDuration="6m44.797477752s" totalImagesPullingTime="6.725466109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:34.796805475 +0000 UTC m=+39.875805445" watchObservedRunningTime="2026-11-01 22:23:34.797477752 +0000 UTC m=+39.876477722"
Nov 01 22:23:34.918071 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:34.917986    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-85ctr" podStartSLOduration=1.809713745 podStartE2EDuration="8.917968997s" totalImagesPullingTime="7.108255252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:34.91611358 +0000 UTC m=+39.995113552" watchObservedRunningTime="2026-11-01 22:23:34.917968997 +0000 UTC m=+39.996968972"
Nov 01 22:23:35.034982 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:35.034879    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j8gn5" podStartSLOduration=34.471705108 podStartE2EDuration="39.034858835s" totalImagesPullingTime="4.563153727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:35.033536362 +0000 UTC m=+40.112536333" watchObservedRunningTime="2026-11-01 22:23:35.034858835 +0000 UTC m=+40.113858805"
Nov 01 22:23:35.261244 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:35.261207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:35.261408 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:35.261321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:35.261408 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:35.261337    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:35.261408 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:35.261403    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:37.261386512 +0000 UTC m=+42.340386460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:35.261579 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:35.261467    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:35.261579 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:35.261516    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:37.261504253 +0000 UTC m=+42.340504204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:36.759316 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:36.759284    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:36.759779 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:36.759366    2583 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:36.759779 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:36.759727    2583 scope.go:122] "RemoveContainer" containerID="d6a6557aeab3afcd4e100f96b6cd67cffa9d11eb2f38f30d74e4cecd52aed739"
Nov 01 22:23:36.759929 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:36.759905    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-m7tdl_openshift-console-operator(b330b73a-8364-478a-9589-1b24674fb30e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" podUID="b330b73a-8364-478a-9589-1b24674fb30e"
Nov 01 22:23:37.280481 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:37.280439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:37.280657 ip-10-0-129-77 kubenswrapper[2583]: I1101 22:23:37.280565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:37.280657 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:37.280592    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:37.280770 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:37.280675    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.280653391 +0000 UTC m=+46.359653344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:37.280770 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:37.280692    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:37.280770 ip-10-0-129-77 kubenswrapper[2583]: E1101 22:23:37.280763    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.280742262 +0000 UTC m=+46.359742226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:38.349244 ip-10-0-129-77 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 22:23:38.359399 ip-10-0-129-77 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 22:23:38.359571 ip-10-0-129-77 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 22:23:38.359635 ip-10-0-129-77 systemd[1]: kubelet.service: Consumed 2.853s CPU time.
Nov 01 22:23:38.373699 ip-10-0-129-77 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 22:23:38.442853 ip-10-0-129-77 kubenswrapper[4837]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:23:38.442853 ip-10-0-129-77 kubenswrapper[4837]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 22:23:38.442853 ip-10-0-129-77 kubenswrapper[4837]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:23:38.442853 ip-10-0-129-77 kubenswrapper[4837]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:23:38.444944 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444863    4837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.444944 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444943    4837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444950    4837 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444954    4837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444957    4837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444960    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444962    4837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444965    4837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444968    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444971    4837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444973    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444976    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444979    4837 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444981    4837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444984    4837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444995    4837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.444998    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445000    4837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445003    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445006    4837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.445019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445009    4837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445011    4837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445014    4837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445017    4837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445020    4837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445023    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445025    4837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445028    4837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445031    4837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445034    4837 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445036    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445061    4837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445064    4837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445067    4837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445070    4837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445072    4837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445075    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445077    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445080    4837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445083    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.445511 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445085    4837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445088    4837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445090    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445093    4837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445096    4837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445098    4837 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445101    4837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445105    4837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445107    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445110    4837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445112    4837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445116    4837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445121    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445125    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445128    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445131    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445135    4837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445138    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445147    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445150    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.446000 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445152    4837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445155    4837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445158    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445160    4837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445163    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445165    4837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445168    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445181    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445183    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445187    4837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445191    4837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445195    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445199    4837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445203    4837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445207    4837 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445210    4837 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445213    4837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445216    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445219    4837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445222    4837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.446508 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445225    4837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445228    4837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445231    4837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445234    4837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445238    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445241    4837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445243    4837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445246    4837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445249    4837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445253    4837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445257    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445260    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445263    4837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445265    4837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445268    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445271    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445274    4837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445277    4837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445282    4837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.446993 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445284    4837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445287    4837 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445290    4837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445292    4837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445295    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445297    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.445299    4837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446106    4837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446113    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446116    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446119    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446122    4837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446125    4837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446129    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446132    4837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446135    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446138    4837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446141    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446144    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446146    4837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.447467 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446149    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446151    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446155    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446157    4837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446160    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446163    4837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446165    4837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446168    4837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446171    4837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446173    4837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446176    4837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446179    4837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446181    4837 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446184    4837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446187    4837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446189    4837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446192    4837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446195    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446197    4837 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.447951 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446200    4837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446203    4837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446206    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446209    4837 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446211    4837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446214    4837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446217    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446219    4837 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446223    4837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446225    4837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446228    4837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446230    4837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446233    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446237    4837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446241    4837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446244    4837 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446247    4837 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446251    4837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446254    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.448438 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446257    4837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446260    4837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446263    4837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446265    4837 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446268    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446271    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446273    4837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446276    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446279    4837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446281    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446285    4837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446287    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446290    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446292    4837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446295    4837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446298    4837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446300    4837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446303    4837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446305    4837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446308    4837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.448909 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446310    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446313    4837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446316    4837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446319    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446321    4837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446324    4837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446326    4837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446329    4837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446331    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446334    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446337    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446340    4837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446343    4837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446346    4837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446348    4837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446351    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446354    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446356    4837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446360    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446362    4837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.449460 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446365    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446367    4837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446370    4837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446373    4837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446375    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446378    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446380    4837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446383    4837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446386    4837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446388    4837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446390    4837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446393    4837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446395    4837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446398    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446401    4837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446962    4837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446968    4837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446972    4837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446975    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.449953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446979    4837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446982    4837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446985    4837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446987    4837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446990    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446993    4837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.446997    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447000    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447002    4837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447005    4837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447008    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447010    4837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447013    4837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447015    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447018    4837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447021    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447024    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447027    4837 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447030    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447032    4837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.450434 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447035    4837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447037    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447051    4837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447054    4837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447057    4837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447060    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447062    4837 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447065    4837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447068    4837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447071    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447074    4837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447077    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447079    4837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447082    4837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447084    4837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447087    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447089    4837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447092    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447095    4837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.450953 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447097    4837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447100    4837 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447102    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447105    4837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447108    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447110    4837 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447113    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447115    4837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447118    4837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447121    4837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447124    4837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447127    4837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447130    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447133    4837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447135    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447138    4837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447141    4837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447143    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447146    4837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447149    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.451421 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447152    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447154    4837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447157    4837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447160    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447163    4837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447165    4837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447168    4837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447170    4837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447173    4837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447176    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447178    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447181    4837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447184    4837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447186    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447189    4837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447191    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447194    4837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447196    4837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447199    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.451904 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447207    4837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447211    4837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447216    4837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447220    4837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447235    4837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447239    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447242    4837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447245    4837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447249    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447251    4837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447254    4837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447257    4837 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447260    4837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447262    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447265    4837 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447267    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447271    4837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447274    4837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447276    4837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.452614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447279    4837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447282    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447284    4837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447286    4837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.447289    4837 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447394    4837 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447401    4837 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447408    4837 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447412    4837 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447417    4837 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447420    4837 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447425    4837 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447429    4837 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447435    4837 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447438    4837 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447442    4837 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447445    4837 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447448    4837 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447451    4837 flags.go:64] FLAG: --cgroup-root=""
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447456    4837 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447459    4837 flags.go:64] FLAG: --client-ca-file=""
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447462    4837 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447466    4837 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447471    4837 flags.go:64] FLAG: --cluster-domain=""
Nov 01 22:23:38.453330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447474    4837 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447477    4837 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447480    4837 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447484    4837 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447488    4837 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447491    4837 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447494    4837 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447498    4837 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447501    4837 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447505    4837 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447508    4837 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447511    4837 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447514    4837 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447518    4837 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447521    4837 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447524    4837 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447527    4837 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447530    4837 flags.go:64] FLAG: --enable-server="true"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447533    4837 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447538    4837 flags.go:64] FLAG: --event-burst="100"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447541    4837 flags.go:64] FLAG: --event-qps="50"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447545    4837 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447548    4837 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447551    4837 flags.go:64] FLAG: --eviction-hard=""
Nov 01 22:23:38.454012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447555    4837 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447559    4837 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447562    4837 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447567    4837 flags.go:64] FLAG: --eviction-soft=""
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447570    4837 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447573    4837 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447576    4837 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447579    4837 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447581    4837 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447584    4837 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447587    4837 flags.go:64] FLAG: --feature-gates=""
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447591    4837 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447594    4837 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447598    4837 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447601    4837 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447606    4837 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447609    4837 flags.go:64] FLAG: --help="false"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447612    4837 flags.go:64] FLAG: --hostname-override="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447615    4837 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447619    4837 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447622    4837 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447625    4837 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447629    4837 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 22:23:38.454646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447632    4837 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447635    4837 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447638    4837 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447641    4837 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447645    4837 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447648    4837 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447651    4837 flags.go:64] FLAG: --kube-reserved=""
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447654    4837 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447657    4837 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447660    4837 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447663    4837 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447666    4837 flags.go:64] FLAG: --lock-file=""
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447670    4837 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447673    4837 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447676    4837 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447682    4837 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447685    4837 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447688    4837 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447692    4837 flags.go:64] FLAG: --logging-format="text"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447695    4837 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447698    4837 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447701    4837 flags.go:64] FLAG: --manifest-url=""
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447705    4837 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447709    4837 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447712    4837 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 22:23:38.455466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447717    4837 flags.go:64] FLAG: --max-pods="110"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447720    4837 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447723    4837 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447726    4837 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447729    4837 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447732    4837 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447735    4837 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447738    4837 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447745    4837 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447749    4837 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447752    4837 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447755    4837 flags.go:64] FLAG: --pod-cidr=""
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447757    4837 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447760    4837 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447763    4837 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447766    4837 flags.go:64] FLAG: --port="10250"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447769    4837 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447772    4837 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-02843da5daff69353"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447775    4837 flags.go:64] FLAG: --qos-reserved=""
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447780    4837 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447783    4837 flags.go:64] FLAG: --register-node="true"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447786    4837 flags.go:64] FLAG: --register-with-taints=""
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447790    4837 flags.go:64] FLAG: --registry-burst="10"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447793    4837 flags.go:64] FLAG: --registry-qps="5"
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447796    4837 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447799    4837 flags.go:64] FLAG: --reserved-memory=""
Nov 01 22:23:38.456337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447803    4837 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447806    4837 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447809    4837 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447812    4837 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447815    4837 flags.go:64] FLAG: --runonce="false"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447818    4837 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447821    4837 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447824    4837 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447827    4837 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447830    4837 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447833    4837 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447837    4837 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447840    4837 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447843    4837 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447846    4837 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447848    4837 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447851    4837 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447854    4837 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447857    4837 flags.go:64] FLAG: --system-cgroups=""
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447860    4837 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447865    4837 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447869    4837 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447872    4837 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447876    4837 flags.go:64] FLAG: --tls-min-version=""
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447879    4837 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 22:23:38.457103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447883    4837 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447886    4837 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447889    4837 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447892    4837 flags.go:64] FLAG: --v="2"
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447897    4837 flags.go:64] FLAG: --version="false"
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447901    4837 flags.go:64] FLAG: --vmodule=""
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447905    4837 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.447908    4837 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448066    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448071    4837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448075    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448078    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448080    4837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448085    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448087    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448090    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448093    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448096    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448099    4837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448102    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448104    4837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.457745 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448107    4837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448110    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448112    4837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448115    4837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448117    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448120    4837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448123    4837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448125    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448128    4837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448131    4837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448133    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448138    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448140    4837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448143    4837 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448146    4837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448148    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448151    4837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448154    4837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448156    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448159    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.458432 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448162    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448164    4837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448167    4837 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448170    4837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448174    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448177    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448180    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448183    4837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448185    4837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448188    4837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448191    4837 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448193    4837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448196    4837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448199    4837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448203    4837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448207    4837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448210    4837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448213    4837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448216    4837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448219    4837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.458999 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448221    4837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448224    4837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448226    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448231    4837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448233    4837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448236    4837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448239    4837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448241    4837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448244    4837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448248    4837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448252    4837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448255    4837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448258    4837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448260    4837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448263    4837 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448266    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448270    4837 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448272    4837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448275    4837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448278    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.459517 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448281    4837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448283    4837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448286    4837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448289    4837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448291    4837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448294    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448296    4837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448299    4837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448302    4837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448304    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448307    4837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448309    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448312    4837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448315    4837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448318    4837 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448322    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448324    4837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448327    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448330    4837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448332    4837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.460019 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448335    4837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448338    4837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448340    4837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448343    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448345    4837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448348    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448351    4837 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448353    4837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448358    4837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448361    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448363    4837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448366    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448368    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.448394    4837 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448430    4837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.460546 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448435    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448438    4837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448441    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448443    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448446    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448449    4837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448452    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448455    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448458    4837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448460    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448463    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448465    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448470    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448472    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448475    4837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448478    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448480    4837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448483    4837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448486    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.461026 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448488    4837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448491    4837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448494    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448496    4837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448499    4837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448501    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448504    4837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448508    4837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448510    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448513    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448515    4837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448518    4837 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448521    4837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448524    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448527    4837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448529    4837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448532    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448535    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448537    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448540    4837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.461613 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448543    4837 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448546    4837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448548    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448551    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448553    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448557    4837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448560    4837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448562    4837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448565    4837 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448568    4837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448570    4837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448573    4837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448577    4837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448581    4837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448584    4837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448588    4837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448591    4837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448594    4837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448596    4837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.462194 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448601    4837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448603    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448606    4837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448609    4837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448612    4837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448615    4837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448618    4837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448621    4837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448625    4837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448628    4837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448631    4837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448634    4837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448637    4837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448639    4837 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448642    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448645    4837 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448647    4837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448650    4837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448654    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448657    4837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.462661 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448660    4837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448663    4837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448666    4837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448668    4837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448671    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448673    4837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448676    4837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448678    4837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448681    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448684    4837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448686    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448689    4837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448693    4837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448696    4837 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448698    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448701    4837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448704    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448706    4837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448709    4837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448712    4837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.463160 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448714    4837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448717    4837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448720    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448722    4837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448725    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448728    4837 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.448731    4837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.453342    4837 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.453379    4837 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.453424    4837 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.453433    4837 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453687    4837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453698    4837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453701    4837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453704    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453707    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453710    4837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453713    4837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453715    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453718    4837 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.463716 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453720    4837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453723    4837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453726    4837 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453728    4837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453731    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453736    4837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453740    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453743    4837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453746    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453749    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453752    4837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453754    4837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453757    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453760    4837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453763    4837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453766    4837 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453768    4837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453771    4837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453773    4837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453776    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.464218 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453779    4837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453784    4837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453787    4837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453790    4837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453793    4837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453796    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453798    4837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453801    4837 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453804    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453807    4837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453809    4837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453812    4837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453817    4837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453819    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453823    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453827    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453829    4837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453832    4837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453835    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.464721 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453838    4837 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453840    4837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453843    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453846    4837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453849    4837 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453851    4837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453854    4837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453857    4837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453859    4837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453862    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453864    4837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453867    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453870    4837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453873    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453875    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453878    4837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453881    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453884    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453886    4837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.465308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453889    4837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453892    4837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453894    4837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453897    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453900    4837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453902    4837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453905    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453908    4837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453910    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453913    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453916    4837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453918    4837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453921    4837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453924    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453926    4837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453929    4837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453931    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453934    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453936    4837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.465761 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453939    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453941    4837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453944    4837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453947    4837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453949    4837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453952    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453954    4837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453957    4837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453959    4837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453962    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453966    4837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453968    4837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453971    4837 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453973    4837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453976    4837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453978    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453981    4837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453983    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453986    4837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.466342 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.453988    4837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.454016    4837 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454190    4837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454198    4837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454201    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454204    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454207    4837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454211    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454214    4837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454216    4837 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454219    4837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454222    4837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454224    4837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454227    4837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454230    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454232    4837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.466929 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454235    4837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454237    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454240    4837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454243    4837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454245    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454248    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454251    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454253    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454258    4837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454262    4837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454265    4837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454268    4837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454271    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454274    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454276    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454279    4837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454281    4837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454284    4837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454286    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.467494 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454289    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454292    4837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454295    4837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454297    4837 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454300    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454303    4837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454306    4837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454309    4837 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454311    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454314    4837 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454316    4837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454319    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454322    4837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454324    4837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454327    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454329    4837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454332    4837 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454334    4837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454337    4837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454341    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.468077 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454343    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454347    4837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454351    4837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454354    4837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454356    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454359    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454362    4837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454365    4837 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454367    4837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454370    4837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454372    4837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454375    4837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454377    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454380    4837 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454383    4837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454385    4837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454387    4837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454390    4837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454393    4837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.468552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454396    4837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454398    4837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454401    4837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454404    4837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454406    4837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454409    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454411    4837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454414    4837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454417    4837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454420    4837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454422    4837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454425    4837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454428    4837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454430    4837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454433    4837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454435    4837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454438    4837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454441    4837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454443    4837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454446    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.469008 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454449    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454451    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454454    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454456    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454459    4837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454461    4837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454464    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454466    4837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454469    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454471    4837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454474    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454476    4837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454480    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454483    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.454506    4837 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:23:38.469495 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454540    4837 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454545    4837 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454547    4837 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454550    4837 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454554    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454557    4837 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454560    4837 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454562    4837 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454565    4837 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454568    4837 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454570    4837 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454573    4837 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454575    4837 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454578    4837 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454580    4837 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454583    4837 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454586    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454588    4837 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454591    4837 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454593    4837 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.469949 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454596    4837 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454598    4837 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454601    4837 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454603    4837 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454606    4837 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454608    4837 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454611    4837 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454613    4837 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454616    4837 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454618    4837 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454622    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454625    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454627    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454630    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454632    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454635    4837 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454638    4837 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454640    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454643    4837 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.470524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454647    4837 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454651    4837 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454656    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454659    4837 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454663    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454667    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454671    4837 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454675    4837 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454679    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454683    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454687    4837 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454691    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454695    4837 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454700    4837 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454703    4837 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454718    4837 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454722    4837 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454726    4837 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454731    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454735    4837 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.471114 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454739    4837 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454743    4837 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454747    4837 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454761    4837 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454765    4837 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454768    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454771    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454774    4837 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454777    4837 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454782    4837 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454784    4837 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454787    4837 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454790    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454792    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454795    4837 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454797    4837 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454800    4837 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454803    4837 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454806    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.471996 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454809    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454812    4837 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454814    4837 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454817    4837 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454820    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454822    4837 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454825    4837 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454829    4837 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454831    4837 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454834    4837 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454836    4837 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454839    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454841    4837 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454844    4837 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454846    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454849    4837 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454851    4837 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454854    4837 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454857    4837 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454860    4837 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.472731 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454862    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454866    4837 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454870    4837 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454872    4837 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454875    4837 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454878    4837 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454882    4837 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:38.454887    4837 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.454943    4837 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.455984    4837 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.456058    4837 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.456240    4837 server.go:1014] "Starting client certificate rotation"
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.456389    4837 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.456452    4837 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:56 +0000 UTC" deadline="2028-04-20 02:31:34.320865109 +0000 UTC"
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.456493    4837 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12844h7m55.864375711s"
Nov 01 22:23:38.473635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.456687    4837 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:23:38.474403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.457844    4837 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:23:38.474403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.458966    4837 log.go:25] "Validated CRI v1 runtime API"
Nov 01 22:23:38.474403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.460610    4837 log.go:25] "Validated CRI v1 image API"
Nov 01 22:23:38.474403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.461332    4837 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 22:23:38.474403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.463148    4837 fs.go:135] Filesystem UUIDs: map[57d88cbb-d85f-4a69-b760-ef0af37efad2:/dev/nvme0n1p3 5f1f435a-d8d8-40df-837b-767ead34068d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 01 22:23:38.474669 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.463170    4837 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b59a9fe19dd0a023d76690f99f29fa3a61e7e204f9ea5e0c01832ef4279a428/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b59a9fe19dd0a023d76690f99f29fa3a61e7e204f9ea5e0c01832ef4279a428/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10509ed8e2ff1cfb0c40c5ac8beb121b55988d2a5e82026c10d42005b226c224/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10509ed8e2ff1cfb0c40c5ac8beb121b55988d2a5e82026c10d42005b226c224/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15e949fd7bd165ff9c0b11d52e4c5441b857da2069d86eb8d9e8b6f3a50ab0df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15e949fd7bd165ff9c0b11d52e4c5441b857da2069d86eb8d9e8b6f3a50ab0df/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1785ae234ce8733a047c2ff292d6ed523f1fa5520de77818602bb9dc8625c43b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1785ae234ce8733a047c2ff292d6ed523f1fa5520de77818602bb9dc8625c43b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d353c680b6240a3b8d6d689c0cfc5933b31575176ef6e8f002e32bf6aab4601/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d353c680b6240a3b8d6d689c0cfc5933b31575176ef6e8f002e32bf6aab4601/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32b36eebd80dd0f63ac38ffa798818d7e1dbc146597b87182b0479658ae55d21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32b36eebd80dd0f63ac38ffa798818d7e1dbc146597b87182b0479658ae55d21/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47ba8d729fb8b117d87076fd111776996216e9d31fa46b903b5481ecc3ccfbf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47ba8d729fb8b117d87076fd111776996216e9d31fa46b903b5481ecc3ccfbf2/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58c85b4f95a81e472be585d174fd402e60520afe34b1d9d1e36738e974253e52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58c85b4f95a81e472be585d174fd402e60520afe34b1d9d1e36738e974253e52/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a0790919b718dbf72ffd4c0f5103bc6c863366d715bb1e4d57edd6a3fd1aa07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a0790919b718dbf72ffd4c0f5103bc6c863366d715bb1e4d57edd6a3fd1aa07/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cfdd2ef206e9e1cf12d1d885bc6d38cfbb51e77c76cc977545a1e440c0ecf33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cfdd2ef206e9e1cf12d1d885bc6d38cfbb51e77c76cc977545a1e440c0ecf33/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dc5e9b7dc6836b44787ec04bf937d6a43d1d180466bda7c0994ac18a61df6d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dc5e9b7dc6836b44787ec04bf937d6a43d1d180466bda7c0994ac18a61df6d0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b67db994940be8f8db241632ef45eda61a359574bd4cc5631ee23d585a73aa2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b67db994940be8f8db241632ef45eda61a359574bd4cc5631ee23d585a73aa2/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bb726e0f0e75ba6745e197e0a3b15d44ee248e7453921cb6718acee501f9e49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bb726e0f0e75ba6745e197e0a3b15d44ee248e7453921cb6718acee501f9e49/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d0f28e49f9a66ba975f5344a1ef326078b77cf1165f1f1d8701baa492a633fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d0f28e49f9a66ba975f5344a1ef326078b77cf1165f1f1d8701baa492a633fb/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a38dd3de3f5428a992878a5f1958ee3b87f27e82d339c706feed24fc8ae60c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a38dd3de3f5428a992878a5f1958ee3b87f27e82d339c706feed24fc8ae60c3/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b9aa06f8440a40b1b1675a89c33fda35b40278dfec730e2843f55df97eff43c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b9aa06f8440a40b1b1675a89c33fda35b40278dfec730e2843f55df97eff43c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa90f3d160e5d852429c0d770f93d61fcedb887795c49d8d257fbe55bb1aed94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa90f3d160e5d852429c0d770f93d61fcedb887795c49d8d257fbe55bb1aed94/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6b0092890cfb3ab82ce07038faf4ef3d278826083f772b83dc7b173a4956867/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6b0092890cfb3ab82ce07038faf4ef3d278826083f772b83dc7b173a4956867/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e990ee4370f22529169c6c86e23a3b211ac266abef663ecdf16744491f83726d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e990ee4370f22529169c6c86e23a3b211ac266abef663ecdf16744491f83726d/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f452eab25f548ccb562744ba964cd762f904182a42a649a2ca173d3363930467/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f452eab25f548ccb562744ba964cd762f904182a42a649a2ca173d3363930467/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f669084e6e5b5a29626cbbe2e84e65a261caf34147b8c1e0f8fe4f1fc92fad8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f669084e6e5b5a29626cbbe2e84e65a261caf34147b8c1e0f8fe4f1fc92fad8d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb69500a3ffaad1b5b2997ea7613c5fa7e69c4d874127a8c77fc8060ffa26fa1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb69500a3ffaad1b5b2997ea7613c5fa7e69c4d874127a8c77fc8060ffa26fa1/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1878f36c-168f-490a-bfef-bbe0f477bf88/volumes/kubernetes.io~projected/kube-api-access-qjmv5:{mountpoint:/var/lib/kubelet/pods/1878f36c-168f-490a-bfef-bbe0f477bf88/volumes/kubernetes.io~projected/kube-api-access-qjmv5 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~projected/kube-api-access-65zn5:{mountpoint:/var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~projected/kube-api-access-65zn5 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~secret/default-certificate major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~secret/stats-auth major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/244d4caf-546f-46aa-bdf2-8023694c22a1/volumes/kubernetes.io~projected/kube-api-access-m7wjk:{mountpoint:/var/lib/kubelet/pods/244d4caf-546f-46aa-bdf2-8023694c22a1/volumes/kubernetes.io~projected/kube-api-access-m7wjk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28838384-0663-4a33-9903-26afb7288d9e/volumes/kubernetes.io~projected/kube-api-access-6m46p:{mountpoint:/var/lib/kubelet/pods/28838384-0663-4a33-9903-26afb7288d9e/volumes/kubernetes.io~projected/kube-api-access-6m46p major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ee57462-5ba3-4cfe-a233-13c046a7e797/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2ee57462-5ba3-4cfe-a233-13c046a7e797/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f63221e-2909-4490-bb43-cf2a6164d44c/volumes/kubernetes.io~projected/kube-api-access-r44ms:{mountpoint:/var/lib/kubelet/pods/2f63221e-2909-4490-bb43-cf2a6164d44c/volumes/kubernetes.io~projected/kube-api-access-r44ms major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46fb3ebe-f172-40f8-ba72-3a2370cb447a/volumes/kubernetes.io~projected/kube-api-access-rl7r2:{mountpoint:/var/lib/kubelet/pods/46fb3ebe-f172-40f8-ba72-3a2370cb447a/volumes/kubernetes.io~projected/kube-api-access-rl7r2 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/540ce336-6599-4411-bfc4-a475adfc2ecb/volumes/kubernetes.io~projected/kube-api-access-86rtr:{mountpoint:/var/lib/kubelet/pods/540ce336-6599-4411-bfc4-a475adfc2ecb/volumes/kubernetes.io~projected/kube-api-access-86rtr major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b6492e2-6280-45e6-80df-537fa2489fdf/volumes/kubernetes.io~projected/kube-api-access-xtxd8:{mountpoint:/var/lib/kubelet/pods/5b6492e2-6280-45e6-80df-537fa2489fdf/volumes/kubernetes.io~projected/kube-api-access-xtxd8 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b6492e2-6280-45e6-80df-537fa2489fdf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5b6492e2-6280-45e6-80df-537fa2489fdf/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~projected/kube-api-access-hnp77:{mountpoint:/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~projected/kube-api-access-hnp77 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6854396e-1bc7-4bab-8cd7-68d758d7d74f/volumes/kubernetes.io~projected/kube-api-access-hxp7p:{mountpoint:/var/lib/kubelet/pods/6854396e-1bc7-4bab-8cd7-68d758d7d74f/volumes/kubernetes.io~projected/kube-api-access-hxp7p major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93e7ee65-e872-4029-b310-a6f04d32c8fd/volumes/kubernetes.io~projected/kube-api-access-ttwbz:{mountpoint:/var/lib/kubelet/pods/93e7ee65-e872-4029-b310-a6f04d32c8fd/volumes/kubernetes.io~projected/kube-api-access-ttwbz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/990ac23b-9dab-47c9-a8bf-13907483a733/volumes/kubernetes.io~projected/kube-api-access-nhfc7:{mountpoint:/var/lib/kubelet/pods/990ac23b-9dab-47c9-a8bf-13907483a733/volumes/kubernetes.io~projected/kube-api-access-nhfc7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ced8467-be14-4a8a-a3ae-15566f579cc2/volumes/kubernetes.io~projected/kube-api-access-vq2mb:{mountpoint:/var/lib/kubelet/pods/9ced8467-be14-4a8a-a3ae-15566f579cc2/volumes/kubernetes.io~projected/kube-api-access-vq2mb major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ced8467-be14-4a8a-a3ae-15566f579cc2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9ced8467-be14-4a8a-a3ae-15566f579cc2/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~projected/kube-api-access-mds7f:{mountpoint:/var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~projected/kube-api-access-mds7f major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volumes/kubernetes.io~projected/kube-api-access-6x6dp:{mountpoint:/var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volumes/kubernetes.io~projected/kube-api-access-6x6dp major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b330b73a-8364-478a-9589-1b24674fb30e/volumes/kubernetes.io~projected/kube-api-access-qpmbj:{mountpoint:/var/lib/kubelet/pods/b330b73a-8364-478a-9589-1b24674fb30e/volumes/kubernetes.io~projected/kube-api-access-qpmbj major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b330b73a-8364-478a-9589-1b24674fb30e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b330b73a-8364-478a-9589-1b24674fb30e/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb0df7fd-8362-4c2a-8258-e79901e6c402/volumes/kubernetes.io~projected/kube-api-access-tvqct:{mountpoint:/var/lib/kubelet/pods/bb0df7fd-8362-4c2a-8258-e79901e6c402/volumes/kubernetes.io~projected/kube-api-access-tvqct major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c026e341-04cc-4a50-bb3e-1ff663c2c7e9/volumes/kubernetes.io~projected/kube-api-access-nn6jq:{mountpoint:/var/lib/kubelet/pods/c026e341-04cc-4a50-bb3e-1ff663c2c7e9/volumes/kubernetes.io~projected/kube-api-access-nn6jq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3e0e60c-b299-48da-86b6-3cf0dfdfa238/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c3e0e60c-b299-48da-86b6-3cf0dfdfa238/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9679225-9a70-433b-876a-6bd268ae64e4/volumes/kubernetes.io~projected/kube-api-access-wl9z9:{mountpoint:/var/lib/kubelet/pods/c9679225-9a70-433b-876a-6bd268ae64e4/volumes/kubernetes.io~projected/kube-api-access-wl9z9 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9679225-9a70-433b-876a-6bd268ae64e4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c9679225-9a70-433b-876a-6bd268ae64e4/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbfec945-9ea9-4a51-a7fd-55714dfa01a6/volumes/kubernetes.io~projected/kube-api-access-9sgbc:{mountpoint:/var/lib/kubelet/pods/cbfec945-9ea9-4a51-a7fd-55714dfa01a6/volumes/kubernetes.io~projected/kube-api-access-9sgbc major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbfec945-9ea9-4a51-a7fd-55714dfa01a6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cbfec945-9ea9-4a51-a7fd-55714dfa01a6/volumes/kubernetes.io~secret/cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce89b463-4035-4c17-adc5-4a3e1c07e30b/volumes/kubernetes.io~projected/kube-api-access-cv7rv:{mountpoint:/var/lib/kubelet/pods/ce89b463-4035-4c17-adc5-4a3e1c07e30b/volumes/kubernetes.io~projected/kube-api-access-cv7rv major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2/volumes/kubernetes.io~projected/kube-api-access-9xmv7:{mountpoint:/var/lib/kubelet/pods/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2/volumes/kubernetes.io~projected/kube-api-access-9xmv7 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d41ea86a-a166-44b7-a8da-2a7512c9e786/volumes/kubernetes.io~projected/kube-api-access-t44kq:{mountpoint:/var/lib/kubelet/pods/d41ea86a-a166-44b7-a8da-2a7512c9e786/volumes/kubernetes.io~projected/kube-api-access-t44kq major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d66bf75c-5104-4918-b1f5-48fc89bf7f73/volumes/kubernetes.io~projected/kube-api-access-c8kl5:{mountpoint:/var/lib/kubelet/pods/d66bf75c-5104-4918-b1f5-48fc89bf7f73/volumes/kubernetes.io~projected/kube-api-access-c8kl5 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc731cae-588c-4d82-850e-1c3c3f954833/volumes/kubernetes.io~projected/kube-api-access-l42kc:{mountpoint:/var/lib/kubelet/pods/dc731cae-588c-4d82-850e-1c3c3f954833/volumes/kubernetes.io~projected/kube-api-access-l42kc major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40/volumes/kubernetes.io~projected/kube-api-access-tds8r:{mountpoint:/var/lib/kubelet/pods/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40/volumes/kubernetes.io~projected/kube-api-access-tds8r major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~projected/kube-api-access-9rnkz:{mountpoint:/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~projected/kube-api-access-9rnkz major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/48c59eaa9228be353592d8c0529c39a7dfde023fc815935ea1933d441b68556e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/71ea62c8211b7c2f898369877d6eac328b6f825c6950639b44d6cebbca3f3197/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3ea98060496faafba9929e26d3340d521d2149bd573bfd9d46239d37241e4fd1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/680385b0ec7dd06e3c16a64a7f089bd5efaecd234e51b345da0a310001411668/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/6f24169ae3f1dcd499043c5ca5d417b119332b28af94fc2eede8f28129a0082e/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/16e30fea2b826eb2abb9cb169a8756b016d3e8353e5c1ecae7626d9519ee9351/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d7f7d03184b04a28e1b725722de4e2f10dbcd4ffe69926551680aa2f73f0c720/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/30f43e0d4df39c528abed39a34d5c9c091c153a79460231ce749987629b1d90f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/51ad56bfc004968aafb83714174292f915a9e72839f9c5952512eaf265c830fd/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/6952df8f2326fa5f1d319b83589938ffd88231155159f8f54190ed754d73eed5/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/c6577110ad7ccb38381c36b429eb791e294b58699e4d06a955f6865159cda8f5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/be2ec87c9e225dbb0520278134ae0752bcf9321f9c909ca62e8f0f9f900c61ac/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/f6abe3df7ceb53342c39bd11285e1e7355107ef8e87f5b98ce1dcf5b80108eb3/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/bacf0509258a8a591a847b501359cf02bdba83fd6239354b639e15629050cd1e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/08bb3a37bd461d4d207ee953233ba28ad08521e96a4db7b19ded07192afdc9fe/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a69caf7c9e9ce684db3442f303b0ca0fe93668e08bd186fa15fea3ee319629c7/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e9bbd7ee3f9f9c015eb99bc092203f0bd78ec32ee82638f30da6c733ab3aae1b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2440fe28bea93240263239eadcff4de4a4ef7e372d0ea37dca7157e0e269ebc8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/51e0596aaa5430096e36054a350578bb3af1a00735cd8293f52db370b148dd65/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/49fe90fe051bb583e1a2104ea29c4935e502b9e7cc569570521ba6cbca1cb497/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/935f2bd84c91de531243b31701de62fdddd0d1bf354a63be0e2ee7ff5db3ccef/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/108d47c65d73fc67bfd4467c48b57b402dca0dcbb570211e4033cabac16d106a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8f6a65f2faca345017d531f0c05a7974f7ebcd50c50c9e997143f985cadc5ff3/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/c31ea0ebe92cb9c6e6d20a30c5484a37a0eb720b8f4085b7ca55578a48285d21/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/20d55754f6771195450fb957bb1e5ca0543481e821049b71978e7c5c96ff808d/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/3043206a4680c9b2c90e0d2586924e63dfd169e44ad5de34c32ac824bd7c9f8f/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/06d86882e3071ac9e3fdd85b57e2119b50826dfc2bb6d55df306c65f56b1908b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/ec7668868b41aecdf9f15126db020deda6ddf8a1d82584abdec360f506cc6d65/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/ea3b124965b068e0eb598df588d9113e95d868df5c3d7299bce6ef755d759e05/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/ee97e0785ce91fb8484334fee669774e00352b3e31fb0de0d48cf2b06a866169/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/a72635108f4e40e80e0427469b993ce82a2025462bc044de8bde438a7e61e87d/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/e69c1821dc31216ca8ecd680071faf88ba7b7b30bc6cdab097d6279a781d376b/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/41a4a4dc4ec124c4d08b82d6c432a7264f5f91f203f579c6fb26046fd2e1d7aa/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/96fa30b9a91d618e783f99109a0c1ad2e2cfb72a9697c36cf97caebbd91937d3/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/bc72a08a884a86ebf0ba92510ef1674f828d3ca8ae81703bd4d85b267bc04e9e/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/066582b7fcbcc43a6202efa69cf4b1cf5d1b96eeae6fc7bb0a8e540033f086bb/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/f1e45c8ad8e52bb6ddd7e4e847736b696bc80773036b32b0ef866d74c65e95d6/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/d0db41139557f5b43b6ecf22967d4511e8639f50acffce8a86ee6cba50acc4ff/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/11fbe23e90294fdc3fa27aa363bc4fe18de4c6b4c95cf5f288e842ebbf80efec/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/786f10eb709d41602d23e34f3fc036a48ed8c613a36daf86cb2c2c9e41722e55/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/71cef8ffd5f6fe9afb76ac59311b90527386e531d36ad329925bd62348c382d3/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/f5ec0e3fe709a0a8fce373636e93970632e58cbd59214d222ae7a558a225c128/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/8f1dd61c0dbe92b9c98964f057afe6f69ee9e0718e6922e2b9802c99dba98482/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/79b8972eb32abfbbfdca8a1d51f3cd908602caa6fcb4dbc18bd81624bd2bcdc6/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8ff48ff567312ff01d87cd908257852aedb294b1d36e8527a4c0851d08f1b735/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0a518b5be5833aff7d4689409150d43f37660cdd4f9296ad2e285e831e0266fd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/faf5506a0af4066a501b8933950ee4f4775b46617fab01c220168c71f10e06a9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/eac193ff866a0460c1ac1e066c9537b3717df849e382b4fd24a9645cec57ec26/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/8e3e6735541697655b01166b12efb567d712bfe037a4cd47f1781c1f49b7c29c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a94583eab84413bbe90140f3dd8ae72cd93be29fe150de0cd7c5641c581459b0/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c2fd086f6a2b409ad0faecbea86c2755fb46fd468b51b0609814dd7b1ca42601/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/12e97db4e6b49b3d4bfd68a9ea956f0d51c86710d4f046f34b38d568af009ae5/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c436a60f9659d92adc5211437935bdb9b28d81107c4d994ae10b39435f844364/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/874ef423ef1f828f2dbe1f133987fd04a30f31b4b6ab705b9ac864225b5eb56e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.470415    4837 manager.go:217] Machine: {Timestamp:2026-11-01 22:23:38.469908817 +0000 UTC m=+0.059107869 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100337 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27b2bf15ce1934842400f14ee02186 SystemUUID:ec27b2bf-15ce-1934-8424-00f14ee02186 BootID:ca26c90c-a004-468b-8b79-1156941d4e46 Filesystems:[{Device:/var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~projected/kube-api-access-mds7f DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce89b463-4035-4c17-adc5-4a3e1c07e30b/volumes/kubernetes.io~projected/kube-api-access-cv7rv DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b6492e2-6280-45e6-80df-537fa2489fdf/volumes/kubernetes.io~projected/kube-api-access-xtxd8 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ced8467-be14-4a8a-a3ae-15566f579cc2/volumes/kubernetes.io~projected/kube-api-access-vq2mb DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a38dd3de3f5428a992878a5f1958ee3b87f27e82d339c706feed24fc8ae60c3/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2/volumes/kubernetes.io~projected/kube-api-access-9xmv7 DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f452eab25f548ccb562744ba964cd762f904182a42a649a2ca173d3363930467/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volumes/kubernetes.io~projected/kube-api-access-6x6dp DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9679225-9a70-433b-876a-6bd268ae64e4/volumes/kubernetes.io~projected/kube-api-access-wl9z9 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1878f36c-168f-490a-bfef-bbe0f477bf88/volumes/kubernetes.io~projected/kube-api-access-qjmv5 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bb726e0f0e75ba6745e197e0a3b15d44ee248e7453921cb6718acee501f9e49/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e990ee4370f22529169c6c86e23a3b211ac266abef663ecdf16744491f83726d/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb69500a3ffaad1b5b2997ea7613c5fa7e69c4d874127a8c77fc8060ffa26fa1/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ced8467-be14-4a8a-a3ae-15566f579cc2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b330b73a-8364-478a-9589-1b24674fb30e/volumes/kubernetes.io~projected/kube-api-access-qpmbj DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b9aa06f8440a40b1b1675a89c33fda35b40278dfec730e2843f55df97eff43c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d41ea86a-a166-44b7-a8da-2a7512c9e786/volumes/kubernetes.io~projected/kube-api-access-t44kq DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bb0df7fd-8362-4c2a-8258-e79901e6c402/volumes/kubernetes.io~projected/kube-api-access-tvqct DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d353c680b6240a3b8d6d689c0cfc5933b31575176ef6e8f002e32bf6aab4601/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~projected/kube-api-access-65zn5 DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9679225-9a70-433b-876a-6bd268ae64e4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/540ce336-6599-4411-bfc4-a475adfc2ecb/volumes/kubernetes.io~projected/kube-api-access-86rtr DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6b0092890cfb3ab82ce07038faf4ef3d278826083f772b83dc7b173a4956867/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93e7ee65-e872-4029-b310-a6f04d32c8fd/volumes/kubernetes.io~projected/kube-api-access-ttwbz DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40/volumes/kubernetes.io~projected/kube-api-access-tds8r DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f669084e6e5b5a29626cbbe2e84e65a261caf34147b8c1e0f8fe4f1fc92fad8d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc731cae-588c-4d82-850e-1c3c3f954833/volumes/kubernetes.io~projected/kube-api-access-l42kc DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b6492e2-6280-45e6-80df-537fa2489fdf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1785ae234ce8733a047c2ff292d6ed523f1fa5520de77818602bb9dc8625c43b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b67db994940be8f8db241632ef45eda61a359574bd4cc5631ee23d585a73aa2/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a9038753-40e0-4e1f-aacc-da2a573be477/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes/kubernetes.io~projected/kube-api-access-9rnkz DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa90f3d160e5d852429c0d770f93d61fcedb887795c49d8d257fbe55bb1aed94/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~projected/kube-api-access-hnp77 DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32b36eebd80dd0f63ac38ffa798818d7e1dbc146597b87182b0479658ae55d21/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ee57462-5ba3-4cfe-a233-13c046a7e797/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/990ac23b-9dab-47c9-a8bf-13907483a733/volumes/kubernetes.io~projected/kube-api-access-nhfc7 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46fb3ebe-f172-40f8-ba72-3a2370cb447a/volumes/kubernetes.io~projected/kube-api-access-rl7r2 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/234fcee4-4ad0-4144-8b1b-7f16e714a08f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b330b73a-8364-478a-9589-1b24674fb30e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/244d4caf-546f-46aa-bdf2-8023694c22a1/volumes/kubernetes.io~projected/kube-api-access-m7wjk DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b0f6996e-a749-41be-9a50-eaa334e1f584/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f63221e-2909-4490-bb43-cf2a6164d44c/volumes/kubernetes.io~projected/kube-api-access-r44ms DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47ba8d729fb8b117d87076fd111776996216e9d31fa46b903b5481ecc3ccfbf2/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6854396e-1bc7-4bab-8cd7-68d758d7d74f/volumes/kubernetes.io~projected/kube-api-access-hxp7p DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c026e341-04cc-4a50-bb3e-1ff663c2c7e9/volumes/kubernetes.io~projected/kube-api-access-nn6jq DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d0f28e49f9a66ba975f5344a1ef326078b77cf1165f1f1d8701baa492a633fb/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10509ed8e2ff1cfb0c40c5ac8beb121b55988d2a5e82026c10d42005b226c224/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58c85b4f95a81e472be585d174fd402e60520afe34b1d9d1e36738e974253e52/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d66bf75c-5104-4918-b1f5-48fc89bf7f73/volumes/kubernetes.io~projected/kube-api-access-c8kl5 DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28838384-0663-4a33-9903-26afb7288d9e/volumes/kubernetes.io~projected/kube-api-access-6m46p DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15e949fd7bd165ff9c0b11d52e4c5441b857da2069d86eb8d9e8b6f3a50ab0df/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dc5e9b7dc6836b44787ec04bf937d6a43d1d180466bda7c0994ac18a61df6d0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbfec945-9ea9-4a51-a7fd-55714dfa01a6/volumes/kubernetes.io~projected/kube-api-access-9sgbc DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b59a9fe19dd0a023d76690f99f29fa3a61e7e204f9ea5e0c01832ef4279a428/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cfdd2ef206e9e1cf12d1d885bc6d38cfbb51e77c76cc977545a1e440c0ecf33/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbfec945-9ea9-4a51-a7fd-55714dfa01a6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3e0e60c-b299-48da-86b6-3cf0dfdfa238/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a0790919b718dbf72ffd4c0f5103bc6c863366d715bb1e4d57edd6a3fd1aa07/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b59a9fe19dd0a0 MacAddress:62:62:26:d6:d9:1e Speed:10000 Mtu:8901} {Name:10509ed8e2ff1cf MacAddress:3a:6b:03:60:2e:93 Speed:10000 Mtu:8901} {Name:15e949fd7bd165f MacAddress:d2:18:d0:a2:dd:69 Speed:10000 Mtu:8901} {Name:32b36eebd80dd0f MacAddress:26:ff:d4:23:6e:2c Speed:10000 Mtu:8901} {Name:47ba8d729fb8b11 MacAddress:8a:2a:c9:3c:46:1d Speed:10000 Mtu:8901} {Name:58c85b4f95a81e4 MacAddress:9a:c7:fb:c1:e1:24 Speed:10000 Mtu:8901} {Name:8bb726e0f0e75ba MacAddress:4e:0b:07:91:1f:af Speed:10000 Mtu:8901} {Name:9a38dd3de3f5428 MacAddress:36:0e:fe:0f:f1:1b Speed:10000 Mtu:8901} {Name:aa90f3d160e5d85 MacAddress:8a:d7:67:b6:eb:b1 Speed:10000 Mtu:8901} {Name:b6b0092890cfb3a MacAddress:d6:44:c9:12:cf:bb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fe:ec:58:6b Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:0e:08:b7:cd:d8 Speed:0 Mtu:8901} {Name:e990ee4370f2252 MacAddress:be:6c:03:ca:21:3c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:fe:ec:58:6b Speed:0 Mtu:9001} {Name:f452eab25f548cc MacAddress:ca:4d:31:e7:75:1f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a2:67:29:7f:3c:b0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:72:2b:6b:f8:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.470761    4837 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.470807    4837 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471184    4837 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471214    4837 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-77.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471415    4837 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471428    4837 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471443    4837 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471461    4837 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471488    4837 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471710    4837 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.471988    4837 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.472065    4837 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.472087    4837 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.472111    4837 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.472127    4837 kubelet.go:405] "Adding apiserver pod source"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.472145    4837 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.473364    4837 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.473378    4837 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:23:38.475020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.474059    4837 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 22:23:38.476831 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.476812    4837 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 22:23:38.476831 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.476826    4837 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:23:38.476956 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.476935    4837 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 22:23:38.477155 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477142    4837 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 22:23:38.477211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477172    4837 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 22:23:38.477273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477257    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 22:23:38.477273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477268    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477278    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477288    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477294    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477301    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477306    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477312    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477318    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477324    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477335    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 22:23:38.477366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477347    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 22:23:38.477832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477373    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 22:23:38.477832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.477383    4837 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 22:23:38.480797 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.480784    4837 server.go:1263] "Started kubelet"
Nov 01 22:23:38.481237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.480842    4837 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 22:23:38.481237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.480883    4837 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 22:23:38.481237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.480955    4837 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 22:23:38.481290 ip-10-0-129-77 systemd[1]: Started Kubernetes Kubelet.
Nov 01 22:23:38.481438 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.481393    4837 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 22:23:38.482178 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.482160    4837 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 22:23:38.483782 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.483766    4837 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.483781    4837 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.483832    4837 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:58 +0000 UTC" deadline="2028-05-28 01:40:46.600778374 +0000 UTC"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.483861    4837 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13755h17m8.116920296s"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.483869    4837 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.483884    4837 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.483882    4837 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.484005    4837 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.487735    4837 factory.go:55] Registering systemd factory
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.487758    4837 factory.go:223] Registration of the systemd container factory successfully
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488604    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" volumeName="kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488640    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0f6996e-a749-41be-9a50-eaa334e1f584" volumeName="kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488664    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbfec945-9ea9-4a51-a7fd-55714dfa01a6" volumeName="kubernetes.io/projected/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-kube-api-access-9sgbc" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488676    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce89b463-4035-4c17-adc5-4a3e1c07e30b" volumeName="kubernetes.io/projected/ce89b463-4035-4c17-adc5-4a3e1c07e30b-kube-api-access-cv7rv" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488693    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc731cae-588c-4d82-850e-1c3c3f954833" volumeName="kubernetes.io/projected/dc731cae-588c-4d82-850e-1c3c3f954833-kube-api-access-l42kc" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488706    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee57462-5ba3-4cfe-a233-13c046a7e797" volumeName="kubernetes.io/configmap/2ee57462-5ba3-4cfe-a233-13c046a7e797-konnectivity-ca" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488722    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0f6996e-a749-41be-9a50-eaa334e1f584" volumeName="kubernetes.io/projected/b0f6996e-a749-41be-9a50-eaa334e1f584-kube-api-access-6x6dp" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488733    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbfec945-9ea9-4a51-a7fd-55714dfa01a6" volumeName="kubernetes.io/secret/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-cert" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488749    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6492e2-6280-45e6-80df-537fa2489fdf" volumeName="kubernetes.io/projected/5b6492e2-6280-45e6-80df-537fa2489fdf-kube-api-access-xtxd8" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488774    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" volumeName="kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488788    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ced8467-be14-4a8a-a3ae-15566f579cc2" volumeName="kubernetes.io/projected/9ced8467-be14-4a8a-a3ae-15566f579cc2-kube-api-access-vq2mb" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488805    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9679225-9a70-433b-876a-6bd268ae64e4" volumeName="kubernetes.io/projected/c9679225-9a70-433b-876a-6bd268ae64e4-kube-api-access-wl9z9" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488817    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce89b463-4035-4c17-adc5-4a3e1c07e30b" volumeName="kubernetes.io/configmap/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-config" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488829    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69ab461-32d6-44ee-912e-95c4c27d6696" volumeName="kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488850    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1878f36c-168f-490a-bfef-bbe0f477bf88" volumeName="kubernetes.io/projected/1878f36c-168f-490a-bfef-bbe0f477bf88-kube-api-access-qjmv5" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488879    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234fcee4-4ad0-4144-8b1b-7f16e714a08f" volumeName="kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-default-certificate" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488892    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2b5233e-d5ac-4fa6-adff-97b78dcd1e40" volumeName="kubernetes.io/projected/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-kube-api-access-tds8r" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488904    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69ab461-32d6-44ee-912e-95c4c27d6696" volumeName="kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488922    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6492e2-6280-45e6-80df-537fa2489fdf" volumeName="kubernetes.io/secret/5b6492e2-6280-45e6-80df-537fa2489fdf-serving-cert" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488935    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="990ac23b-9dab-47c9-a8bf-13907483a733" volumeName="kubernetes.io/projected/990ac23b-9dab-47c9-a8bf-13907483a733-kube-api-access-nhfc7" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488951    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0f6996e-a749-41be-9a50-eaa334e1f584" volumeName="kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-env-overrides" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488962    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e0e60c-b299-48da-86b6-3cf0dfdfa238" volumeName="kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488975    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d41ea86a-a166-44b7-a8da-2a7512c9e786" volumeName="kubernetes.io/projected/d41ea86a-a166-44b7-a8da-2a7512c9e786-kube-api-access-t44kq" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.488990    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2b5233e-d5ac-4fa6-adff-97b78dcd1e40" volumeName="kubernetes.io/configmap/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-serviceca" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489002    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f63221e-2909-4490-bb43-cf2a6164d44c" volumeName="kubernetes.io/empty-dir/2f63221e-2909-4490-bb43-cf2a6164d44c-tmp-dir" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489018    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" volumeName="kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489030    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e7ee65-e872-4029-b310-a6f04d32c8fd" volumeName="kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-daemon-config" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489057    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e7ee65-e872-4029-b310-a6f04d32c8fd" volumeName="kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-cni-binary-copy" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489084    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9679225-9a70-433b-876a-6bd268ae64e4" volumeName="kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-service-ca-bundle" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489100    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf0a7850-9ba3-4124-b0fe-a088bdfd05f2" volumeName="kubernetes.io/projected/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2-kube-api-access-9xmv7" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489112    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="244d4caf-546f-46aa-bdf2-8023694c22a1" volumeName="kubernetes.io/configmap/244d4caf-546f-46aa-bdf2-8023694c22a1-iptables-alerter-script" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489129    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28838384-0663-4a33-9903-26afb7288d9e" volumeName="kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-binary-copy" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489146    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28838384-0663-4a33-9903-26afb7288d9e" volumeName="kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489158    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46fb3ebe-f172-40f8-ba72-3a2370cb447a" volumeName="kubernetes.io/projected/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-api-access-rl7r2" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489176    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a69bb5e-ee33-44a8-912b-cc5e3d017d7b" volumeName="kubernetes.io/configmap/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-nginx-conf" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489188    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ced8467-be14-4a8a-a3ae-15566f579cc2" volumeName="kubernetes.io/configmap/9ced8467-be14-4a8a-a3ae-15566f579cc2-config" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489200    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb0df7fd-8362-4c2a-8258-e79901e6c402" volumeName="kubernetes.io/empty-dir/bb0df7fd-8362-4c2a-8258-e79901e6c402-tmp-dir" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489215    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9679225-9a70-433b-876a-6bd268ae64e4" volumeName="kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489229    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="540ce336-6599-4411-bfc4-a475adfc2ecb" volumeName="kubernetes.io/projected/540ce336-6599-4411-bfc4-a475adfc2ecb-kube-api-access-86rtr" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489245    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" volumeName="kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489259    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9038753-40e0-4e1f-aacc-da2a573be477" volumeName="kubernetes.io/projected/a9038753-40e0-4e1f-aacc-da2a573be477-kube-api-access-mds7f" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489271    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0f6996e-a749-41be-9a50-eaa334e1f584" volumeName="kubernetes.io/secret/b0f6996e-a749-41be-9a50-eaa334e1f584-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489287    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b330b73a-8364-478a-9589-1b24674fb30e" volumeName="kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-trusted-ca" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489299    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026e341-04cc-4a50-bb3e-1ff663c2c7e9" volumeName="kubernetes.io/projected/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kube-api-access-nn6jq" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489315    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f63221e-2909-4490-bb43-cf2a6164d44c" volumeName="kubernetes.io/projected/2f63221e-2909-4490-bb43-cf2a6164d44c-kube-api-access-r44ms" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489328    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46fb3ebe-f172-40f8-ba72-3a2370cb447a" volumeName="kubernetes.io/empty-dir/46fb3ebe-f172-40f8-ba72-3a2370cb447a-data-volume" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489347    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" volumeName="kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489361    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9679225-9a70-433b-876a-6bd268ae64e4" volumeName="kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-snapshots" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489372    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="244d4caf-546f-46aa-bdf2-8023694c22a1" volumeName="kubernetes.io/projected/244d4caf-546f-46aa-bdf2-8023694c22a1-kube-api-access-m7wjk" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489388    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f63221e-2909-4490-bb43-cf2a6164d44c" volumeName="kubernetes.io/configmap/2f63221e-2909-4490-bb43-cf2a6164d44c-config-volume" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489400    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ced8467-be14-4a8a-a3ae-15566f579cc2" volumeName="kubernetes.io/secret/9ced8467-be14-4a8a-a3ae-15566f579cc2-serving-cert" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489414    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e7ee65-e872-4029-b310-a6f04d32c8fd" volumeName="kubernetes.io/projected/93e7ee65-e872-4029-b310-a6f04d32c8fd-kube-api-access-ttwbz" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489428    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b330b73a-8364-478a-9589-1b24674fb30e" volumeName="kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-config" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489440    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d66bf75c-5104-4918-b1f5-48fc89bf7f73" volumeName="kubernetes.io/projected/d66bf75c-5104-4918-b1f5-48fc89bf7f73-kube-api-access-c8kl5" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489457    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69ab461-32d6-44ee-912e-95c4c27d6696" volumeName="kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489468    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" volumeName="kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489244    4837 factory.go:153] Registering CRI-O factory
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489498    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9038753-40e0-4e1f-aacc-da2a573be477" volumeName="kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-tmp" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489524    4837 factory.go:223] Registration of the crio container factory successfully
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489526    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0f6996e-a749-41be-9a50-eaa334e1f584" volumeName="kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-config" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489539    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b330b73a-8364-478a-9589-1b24674fb30e" volumeName="kubernetes.io/secret/b330b73a-8364-478a-9589-1b24674fb30e-serving-cert" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489555    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d41ea86a-a166-44b7-a8da-2a7512c9e786" volumeName="kubernetes.io/configmap/d41ea86a-a166-44b7-a8da-2a7512c9e786-telemetry-config" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489566    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69ab461-32d6-44ee-912e-95c4c27d6696" volumeName="kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489577    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234fcee4-4ad0-4144-8b1b-7f16e714a08f" volumeName="kubernetes.io/projected/234fcee4-4ad0-4144-8b1b-7f16e714a08f-kube-api-access-65zn5" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489583    4837 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489592    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" volumeName="kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489604    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9679225-9a70-433b-876a-6bd268ae64e4" volumeName="kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-tmp" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489621    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69ab461-32d6-44ee-912e-95c4c27d6696" volumeName="kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489633    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b6492e2-6280-45e6-80df-537fa2489fdf" volumeName="kubernetes.io/configmap/5b6492e2-6280-45e6-80df-537fa2489fdf-config" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489644    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b330b73a-8364-478a-9589-1b24674fb30e" volumeName="kubernetes.io/projected/b330b73a-8364-478a-9589-1b24674fb30e-kube-api-access-qpmbj" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489659    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69ab461-32d6-44ee-912e-95c4c27d6696" volumeName="kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489671    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="234fcee4-4ad0-4144-8b1b-7f16e714a08f" volumeName="kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-stats-auth" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489605    4837 factory.go:103] Registering Raw factory
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489687    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28838384-0663-4a33-9903-26afb7288d9e" volumeName="kubernetes.io/projected/28838384-0663-4a33-9903-26afb7288d9e-kube-api-access-6m46p" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489759    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69ab461-32d6-44ee-912e-95c4c27d6696" volumeName="kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489771    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee57462-5ba3-4cfe-a233-13c046a7e797" volumeName="kubernetes.io/secret/2ee57462-5ba3-4cfe-a233-13c046a7e797-agent-certs" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489785    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6854396e-1bc7-4bab-8cd7-68d758d7d74f" volumeName="kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489799    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9038753-40e0-4e1f-aacc-da2a573be477" volumeName="kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-etc-tuned" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489739    4837 manager.go:1199] Started watching for new ooms in manager
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489811    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb0df7fd-8362-4c2a-8258-e79901e6c402" volumeName="kubernetes.io/projected/bb0df7fd-8362-4c2a-8258-e79901e6c402-kube-api-access-tvqct" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489845    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9679225-9a70-433b-876a-6bd268ae64e4" volumeName="kubernetes.io/secret/c9679225-9a70-433b-876a-6bd268ae64e4-serving-cert" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489864    4837 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28838384-0663-4a33-9903-26afb7288d9e" volumeName="kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489876    4837 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.489885    4837 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 22:23:38.491344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.490278    4837 manager.go:319] Starting recovery of all containers
Nov 01 22:23:38.503923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.503905    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/1.log"
Nov 01 22:23:38.505578 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.505556    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/1.log"
Nov 01 22:23:38.515547 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.515502    4837 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 22:23:38.516734 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.516713    4837 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 22:23:38.516734 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.516734    4837 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 22:23:38.516871 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.516755    4837 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 22:23:38.516871 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.516800    4837 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 22:23:38.534309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.534288    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/1.log"
Nov 01 22:23:38.534397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.534320    4837 generic.go:356] "Generic (PLEG): container finished" podID="b330b73a-8364-478a-9589-1b24674fb30e" containerID="d6a6557aeab3afcd4e100f96b6cd67cffa9d11eb2f38f30d74e4cecd52aed739" exitCode=255
Nov 01 22:23:38.541390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.541363    4837 generic.go:356] "Generic (PLEG): container finished" podID="d0a42aef018761dafd58eb8c13dacd2c" containerID="108e6909deb4fcdb3c8015d555de35b0c723f82a802b075c217c97909a221ba4" exitCode=0
Nov 01 22:23:38.557030 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.557006    4837 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="37dd6859334ce20969d23bdcc2f15e31f3be231c8cc97bf2b5821f0385c73851" exitCode=0
Nov 01 22:23:38.557030 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.557027    4837 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="dc8690c2b08fd66e82ecde16cf9647e9cc54173ea4649e39202e8be9a28b86de" exitCode=0
Nov 01 22:23:38.557030 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.557034    4837 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="2ad613449ebaa2a8e08c12e4d8438594c04f2e022f1cbd29cbb8dfc5c5b33dde" exitCode=0
Nov 01 22:23:38.557244 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.557057    4837 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="b16765d00e587e02554ee3c027da4d0c15378971cf750453546e511116c392a2" exitCode=0
Nov 01 22:23:38.557244 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.557064    4837 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="e07f0ffd5ce5ee0638ca44fe20586c14ebb26669881e9c96a6e4b835adfa2d8f" exitCode=0
Nov 01 22:23:38.557244 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.557070    4837 generic.go:356] "Generic (PLEG): container finished" podID="28838384-0663-4a33-9903-26afb7288d9e" containerID="f9e25283528c4029380e4bffea9a4e63ba39a41fc1617693669e07f76ce1993a" exitCode=0
Nov 01 22:23:38.578464 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.578446    4837 manager.go:324] Recovery completed
Nov 01 22:23:38.584012 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.583992    4837 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:23:38.596350 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.596335    4837 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.599117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.599099    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.599223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.599132    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.599223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.599148    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.601625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601568    4837 cpu_manager.go:226] "Starting" policy="none"
Nov 01 22:23:38.601625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601595    4837 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 22:23:38.601625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601610    4837 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 22:23:38.601759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601582    4837 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.601759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601713    4837 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 22:23:38.601759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601723    4837 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 22:23:38.601759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601737    4837 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 22:23:38.601759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601743    4837 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 22:23:38.601759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601750    4837 policy_none.go:50] "Start"
Nov 01 22:23:38.601759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601760    4837 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 22:23:38.601954 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.601772    4837 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 22:23:38.602577 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.602563    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.602626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.602589    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.602626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.602602    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.615557 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.615540    4837 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 22:23:38.615557 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.615556    4837 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 22:23:38.615625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.615563    4837 policy_none.go:44] "Start"
Nov 01 22:23:38.616860 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.616844    4837 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 22:23:38.619735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.619707    4837 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 22:23:38.624165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624152    4837 manager.go:341] "Starting Device Plugin manager"
Nov 01 22:23:38.624243 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.624179    4837 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 22:23:38.624243 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624187    4837 server.go:87] "Starting device plugin registration server"
Nov 01 22:23:38.624348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624336    4837 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 22:23:38.624389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624350    4837 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 22:23:38.624475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624454    4837 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 22:23:38.624619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624570    4837 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 22:23:38.624683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624627    4837 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 22:23:38.624683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624635    4837 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 22:23:38.624777 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.624688    4837 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:38.624617199Z","UUID":"955bff51-d3a7-45c3-b381-12a4ba53a81f","Handler":null,"Name":"","Endpoint":""}
Nov 01 22:23:38.626026 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.625994    4837 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 22:23:38.626026 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.626016    4837 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 22:23:38.626183 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.626022    4837 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 22:23:38.626183 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.626082    4837 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-77.us-west-2.compute.internal\" not found"
Nov 01 22:23:38.672291 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.672275    4837 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 22:23:38.672423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.672407    4837 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:23:38.672911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.672893    4837 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:23:38.673034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.673015    4837 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:23:38.719925 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.719893    4837 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:23:38.725315 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.725300    4837 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.726164 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.726150    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.726228 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.726175    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.726228 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.726185    4837 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-77.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.726228 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.726217    4837 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.816976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.816915    4837 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"]
Nov 01 22:23:38.817514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.817482    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"0dcabca36fe7386806bacef405dbcfa5","Type":"ContainerStarted","Data":"01a9a90ac18f71932de7cab9c04fbfbf11fcc24473dfa7ed8e062d4c958b0014"}
Nov 01 22:23:38.817601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.817523    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"0dcabca36fe7386806bacef405dbcfa5","Type":"ContainerStarted","Data":"8b67db994940be8f8db241632ef45eda61a359574bd4cc5631ee23d585a73aa2"}
Nov 01 22:23:38.817601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.817569    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"d0a42aef018761dafd58eb8c13dacd2c","Type":"ContainerStarted","Data":"fe486dfccbff84f7b7244af7945cb79fcb03690a29401ec8b1ceb437b5cd193e"}
Nov 01 22:23:38.817601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.817590    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"d0a42aef018761dafd58eb8c13dacd2c","Type":"ContainerDied","Data":"108e6909deb4fcdb3c8015d555de35b0c723f82a802b075c217c97909a221ba4"}
Nov 01 22:23:38.817734 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.817608    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal" event={"ID":"d0a42aef018761dafd58eb8c13dacd2c","Type":"ContainerStarted","Data":"1785ae234ce8733a047c2ff292d6ed523f1fa5520de77818602bb9dc8625c43b"}
Nov 01 22:23:38.817734 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.817612    4837 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.817814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.817746    4837 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.829946 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.829932    4837 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.829995 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.829985    4837 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.876089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.876035    4837 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:23:38.876089 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.876082    4837 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.876403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.876389    4837 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:23:38.876470 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:38.876427    4837 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.974605 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.974581    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.974680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.974609    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:38.974680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:38.974626    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dcabca36fe7386806bacef405dbcfa5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"0dcabca36fe7386806bacef405dbcfa5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.075686 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.075661    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.075783 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.075743    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.075783 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.075776    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.075872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.075694    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a42aef018761dafd58eb8c13dacd2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"d0a42aef018761dafd58eb8c13dacd2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.075872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.075814    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dcabca36fe7386806bacef405dbcfa5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"0dcabca36fe7386806bacef405dbcfa5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.075872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.075845    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dcabca36fe7386806bacef405dbcfa5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal\" (UID: \"0dcabca36fe7386806bacef405dbcfa5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.473898 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.473878    4837 apiserver.go:52] "Watching apiserver"
Nov 01 22:23:39.569053 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.569023    4837 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.665195 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.665178    4837 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:23:39.665335 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.665224    4837 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal"
Nov 01 22:23:39.665653 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.665638    4837 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 22:23:39.666262 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.666243    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk","openshift-dns/dns-default-ksjkx","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb","openshift-multus/multus-additional-cni-plugins-zd476","openshift-ingress-canary/ingress-canary-85ctr","openshift-insights/insights-runtime-extractor-h84pf","openshift-multus/multus-r5xnp","openshift-network-diagnostics/network-check-target-j8gn5","openshift-multus/network-metrics-daemon-xvhpg","openshift-insights/insights-operator-7c4986d449-6n7x4","openshift-cluster-node-tuning-operator/tuned-8g2k7","openshift-ingress/router-default-56cdb546f-44l9r","openshift-console-operator/console-operator-7cd56b8dbc-m7tdl","kube-system/konnectivity-agent-hnmh2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-77.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal","openshift-dns/node-resolver-njpz8","openshift-network-console/networking-console-plugin-86c478bb88-ths57","openshift-image-registry/image-registry-d5889b767-99spr","kube-system/kas-connection-checker-685895c7c9-ktgdl","openshift-ovn-kubernetes/ovnkube-node-n2phj","kube-system/kas-connection-checker-685895c7c9-gwk8z","openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx","openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd","openshift-image-registry/node-ca-km7mv","openshift-image-registry/image-registry-677954f946-rrb8k","kube-system/global-pull-secret-syncer-njrpj","openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw","kube-system/kas-connection-checker-685895c7c9-rhtp4","openshift-network-operator/iptables-alerter-tb9n7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"]
Nov 01 22:23:39.670290 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.670271    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:39.670629 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.670616    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.671226 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.671205    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:39.672171 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.672154    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.672888 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.672867    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:39.675267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.675251    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.676088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.676074    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:39.680159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.680145    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.680952 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.680936    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.685770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.685756    4837 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 22:23:39.721738 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.721719    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 22:23:39.721848 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.721837    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 22:23:39.722086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722037    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 22:23:39.722163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722094    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 22:23:39.722163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722108    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 22:23:39.722163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722098    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bn2bj\""
Nov 01 22:23:39.722163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722139    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7fqmk\""
Nov 01 22:23:39.722430 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722270    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 22:23:39.722430 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722328    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 22:23:39.722430 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722330    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 22:23:39.722610 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722452    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gqgtr\""
Nov 01 22:23:39.722840 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722805    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t4c8s\""
Nov 01 22:23:39.722840 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722829    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 22:23:39.722840 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722838    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 22:23:39.723055 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722866    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-25vcq\""
Nov 01 22:23:39.723055 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.722967    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.723162 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723144    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hfmjw\""
Nov 01 22:23:39.723273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723187    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.723273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723237    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.723273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723246    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.723428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723288    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.723428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723318    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.723428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723326    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.723428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723319    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 22:23:39.723428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723346    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.723428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723376    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.723687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723530    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.723687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723557    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 22:23:39.723687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723616    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tdjb6\""
Nov 01 22:23:39.723866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723853    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 22:23:39.723939 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.723895    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.724096 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724057    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8dshl\""
Nov 01 22:23:39.724203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724178    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 22:23:39.724203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724188    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.724203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724178    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 22:23:39.724377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724359    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 22:23:39.724377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724369    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 22:23:39.724461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724424    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.724461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724424    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 22:23:39.724735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724717    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.724787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724775    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 22:23:39.724832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724816    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.724874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724862    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.724923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724865    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 22:23:39.724923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724893    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.724923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724902    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.725078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724905    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.725078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724948    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.725078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.724943    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.725362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.725343    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.725549 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.725529    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2sjxk\""
Nov 01 22:23:39.771147 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.771127    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 22:23:39.771267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.771220    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 22:23:39.771347 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.771310    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 22:23:39.771475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.771458    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 22:23:39.779216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779196    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.779299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779223    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wjk\" (UniqueName: \"kubernetes.io/projected/244d4caf-546f-46aa-bdf2-8023694c22a1-kube-api-access-m7wjk\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:23:39.779299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779240    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-daemon-config\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.779299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779262    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.779449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779343    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.779449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779367    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.779449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779386    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.779449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779402    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-snapshots\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.779449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779417    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-serviceca\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:23:39.779449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779433    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.779449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779439    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.779733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779485    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-snapshots\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.779733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779503    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kl5\" (UniqueName: \"kubernetes.io/projected/d66bf75c-5104-4918-b1f5-48fc89bf7f73-kube-api-access-c8kl5\") pod \"kas-connection-checker-685895c7c9-ktgdl\" (UID: \"d66bf75c-5104-4918-b1f5-48fc89bf7f73\") " pod="kube-system/kas-connection-checker-685895c7c9-ktgdl"
Nov 01 22:23:39.779733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779532    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:39.779733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779658    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.779733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779696    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-stats-auth\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.779733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779720    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-dbus\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:39.779733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779725    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-serviceca\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:23:39.780039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779791    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.780039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779820    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-multus-certs\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.780039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779845    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.780039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779867    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7rv\" (UniqueName: \"kubernetes.io/projected/ce89b463-4035-4c17-adc5-4a3e1c07e30b-kube-api-access-cv7rv\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:39.780039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779882    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-node-log\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.780039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779904    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-stats-auth\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.780039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.779999    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-script-lib\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.780039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780033    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-cert\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780067    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-default-certificate\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780106    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ee57462-5ba3-4cfe-a233-13c046a7e797-konnectivity-ca\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780129    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-cni-binary-copy\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780153    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-cnibin\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780178    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0f6996e-a749-41be-9a50-eaa334e1f584-ovn-node-metrics-cert\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780202    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x6dp\" (UniqueName: \"kubernetes.io/projected/b0f6996e-a749-41be-9a50-eaa334e1f584-kube-api-access-6x6dp\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780238    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-run\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780263    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-lib-modules\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780288    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-default-certificate\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.780358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780290    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmv7\" (UniqueName: \"kubernetes.io/projected/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2-kube-api-access-9xmv7\") pod \"volume-data-source-validator-58c9f5f46d-cwjn4\" (UID: \"cf0a7850-9ba3-4124-b0fe-a088bdfd05f2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780431    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-device-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780454    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/244d4caf-546f-46aa-bdf2-8023694c22a1-iptables-alerter-script\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780474    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2mb\" (UniqueName: \"kubernetes.io/projected/9ced8467-be14-4a8a-a3ae-15566f579cc2-kube-api-access-vq2mb\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780495    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-kubelet-config\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780517    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780545    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f63221e-2909-4490-bb43-cf2a6164d44c-tmp-dir\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780568    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r44ms\" (UniqueName: \"kubernetes.io/projected/2f63221e-2909-4490-bb43-cf2a6164d44c-kube-api-access-r44ms\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780584    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfc7\" (UniqueName: \"kubernetes.io/projected/990ac23b-9dab-47c9-a8bf-13907483a733-kube-api-access-nhfc7\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780621    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-kubernetes\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780683    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-tmp\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780712    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-service-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.780754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780739    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ced8467-be14-4a8a-a3ae-15566f579cc2-config\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780771    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-trusted-ca\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780793    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-tmp\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780797    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rtr\" (UniqueName: \"kubernetes.io/projected/540ce336-6599-4411-bfc4-a475adfc2ecb-kube-api-access-86rtr\") pod \"network-check-source-6f78d59f99-4b4vw\" (UID: \"540ce336-6599-4411-bfc4-a475adfc2ecb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780825    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f63221e-2909-4490-bb43-cf2a6164d44c-tmp-dir\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780840    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-bin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780847    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-service-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780882    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/a9038753-40e0-4e1f-aacc-da2a573be477-kube-api-access-mds7f\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780919    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780936    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780954    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b6492e2-6280-45e6-80df-537fa2489fdf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780991    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqct\" (UniqueName: \"kubernetes.io/projected/bb0df7fd-8362-4c2a-8258-e79901e6c402-kube-api-access-tvqct\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.780996    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ced8467-be14-4a8a-a3ae-15566f579cc2-config\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781010    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781028    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-log-socket\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781061    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781101    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46fb3ebe-f172-40f8-ba72-3a2370cb447a-crio-socket\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781137    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/244d4caf-546f-46aa-bdf2-8023694c22a1-host-slash\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781161    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-slash\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781163    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b6492e2-6280-45e6-80df-537fa2489fdf-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781179    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-var-lib-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781215    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-hostroot\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781239    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781262    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781283    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-etc-kubernetes\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781293    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-trusted-ca\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:39.781292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781299    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7r2\" (UniqueName: \"kubernetes.io/projected/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-api-access-rl7r2\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781320    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781444    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781452    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-multus\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781484    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46fb3ebe-f172-40f8-ba72-3a2370cb447a-data-volume\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781504    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781520    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781544    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781558    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46fb3ebe-f172-40f8-ba72-3a2370cb447a-data-volume\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781563    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmv5\" (UniqueName: \"kubernetes.io/projected/1878f36c-168f-490a-bfef-bbe0f477bf88-kube-api-access-qjmv5\") pod \"kas-connection-checker-685895c7c9-gwk8z\" (UID: \"1878f36c-168f-490a-bfef-bbe0f477bf88\") " pod="kube-system/kas-connection-checker-685895c7c9-gwk8z"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781592    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781639    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781666    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tds8r\" (UniqueName: \"kubernetes.io/projected/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-kube-api-access-tds8r\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781732    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgbc\" (UniqueName: \"kubernetes.io/projected/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-kube-api-access-9sgbc\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781757    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781758    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnp77\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781831    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-kubelet\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781876    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781906    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781939    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781964    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-conf-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.781989    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwbz\" (UniqueName: \"kubernetes.io/projected/93e7ee65-e872-4029-b310-a6f04d32c8fd-kube-api-access-ttwbz\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782014    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782073    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782104    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxd8\" (UniqueName: \"kubernetes.io/projected/5b6492e2-6280-45e6-80df-537fa2489fdf-kube-api-access-xtxd8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782131    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-socket-dir-parent\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782165    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782193    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.782250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782255    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782283    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782299    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ced8467-be14-4a8a-a3ae-15566f579cc2-serving-cert\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782395    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-systemd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782442    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9z9\" (UniqueName: \"kubernetes.io/projected/c9679225-9a70-433b-876a-6bd268ae64e4-kube-api-access-wl9z9\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782476    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-system-cni-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782503    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m46p\" (UniqueName: \"kubernetes.io/projected/28838384-0663-4a33-9903-26afb7288d9e-kube-api-access-6m46p\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782522    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zn5\" (UniqueName: \"kubernetes.io/projected/234fcee4-4ad0-4144-8b1b-7f16e714a08f-kube-api-access-65zn5\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782539    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782547    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9679225-9a70-433b-876a-6bd268ae64e4-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782583    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb0df7fd-8362-4c2a-8258-e79901e6c402-hosts-file\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782600    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-etc-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782616    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782634    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-env-overrides\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782648    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-var-lib-kubelet\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782664    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6492e2-6280-45e6-80df-537fa2489fdf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782849    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ced8467-be14-4a8a-a3ae-15566f579cc2-serving-cert\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782859    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782882    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-config\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782907    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmbj\" (UniqueName: \"kubernetes.io/projected/b330b73a-8364-478a-9589-1b24674fb30e-kube-api-access-qpmbj\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782925    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-k8s-cni-cncf-io\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782941    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-systemd-units\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.782991    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysconfig\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783008    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d41ea86a-a166-44b7-a8da-2a7512c9e786-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783023    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9679225-9a70-433b-876a-6bd268ae64e4-serving-cert\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783066    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783085    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0df7fd-8362-4c2a-8258-e79901e6c402-tmp-dir\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783101    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-netns\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783144    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-kubelet\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783161    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ee57462-5ba3-4cfe-a233-13c046a7e797-agent-certs\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783192    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783206    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-system-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783221    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-netns\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783229    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9679225-9a70-433b-876a-6bd268ae64e4-serving-cert\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783238    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6jq\" (UniqueName: \"kubernetes.io/projected/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kube-api-access-nn6jq\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783255    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783368    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6492e2-6280-45e6-80df-537fa2489fdf-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783413    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783441    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783455    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b330b73a-8364-478a-9589-1b24674fb30e-config\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783472    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-tmp\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783503    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783509    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0df7fd-8362-4c2a-8258-e79901e6c402-tmp-dir\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783551    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783569    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-systemd\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783584    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-cnibin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783604    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783609    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d41ea86a-a166-44b7-a8da-2a7512c9e786-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:39.783596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783582    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9679225-9a70-433b-876a-6bd268ae64e4-tmp\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783715    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-etc-tuned\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783746    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-host\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783764    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783766    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-bin\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783809    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-conf\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783836    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-sys\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783868    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnkz\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.783972    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9038753-40e0-4e1f-aacc-da2a573be477-etc-tuned\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784066    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784088    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f63221e-2909-4490-bb43-cf2a6164d44c-config-volume\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784124    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42kc\" (UniqueName: \"kubernetes.io/projected/dc731cae-588c-4d82-850e-1c3c3f954833-kube-api-access-l42kc\") pod \"kas-connection-checker-685895c7c9-rhtp4\" (UID: \"dc731cae-588c-4d82-850e-1c3c3f954833\") " pod="kube-system/kas-connection-checker-685895c7c9-rhtp4"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784187    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784230    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-netd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784268    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-modprobe-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784292    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784309    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784317    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b330b73a-8364-478a-9589-1b24674fb30e-serving-cert\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784353    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784357    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784448    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784459    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b330b73a-8364-478a-9589-1b24674fb30e-serving-cert\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784530    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t44kq\" (UniqueName: \"kubernetes.io/projected/d41ea86a-a166-44b7-a8da-2a7512c9e786-kube-api-access-t44kq\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784647    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-ovn\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784703    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-config\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.785459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.784771    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-host\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.827638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.827618    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 22:23:39.827638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.827632    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 22:23:39.827744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.827645    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.830888 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.830873    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0f6996e-a749-41be-9a50-eaa334e1f584-ovn-node-metrics-cert\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.840156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840136    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 22:23:39.840247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840175    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 22:23:39.840247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840190    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.840367 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840312    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 22:23:39.840486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840474    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 22:23:39.840623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840599    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 22:23:39.840740 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840725    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 22:23:39.840740 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840729    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.840870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840747    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.840870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840817    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.840870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840821    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 22:23:39.840870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840826    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.841066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840896    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.841066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840898    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 22:23:39.841066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840944    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.841066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.840991    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.841066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841022    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841069    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841092    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jbqbs\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841125    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841140    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841166    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841188    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841231    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmcrl\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841258    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841303    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm5kv\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841269    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841347    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwhsh\""
Nov 01 22:23:39.841370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841365    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841407    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841425    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841434    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-67p2k\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841482    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841581    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841591    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n4xsx\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841597    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841606    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841644    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftrlf\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841663    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hb4ll\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841667    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w2xg9\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841648    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jl5pd\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841668    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wj42\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841683    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j6ll7\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841694    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb5v5\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841703    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ljglf\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841724    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dhrld\""
Nov 01 22:23:39.841914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.841764    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 22:23:39.842911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.842897    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-env-overrides\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.844141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.844124    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ee57462-5ba3-4cfe-a233-13c046a7e797-agent-certs\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:39.845157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.845143    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-config\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.845216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.845172    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f63221e-2909-4490-bb43-cf2a6164d44c-config-volume\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.850404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.850383    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ee57462-5ba3-4cfe-a233-13c046a7e797-konnectivity-ca\") pod \"konnectivity-agent-hnmh2\" (UID: \"2ee57462-5ba3-4cfe-a233-13c046a7e797\") " pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:39.850473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.850405    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-cert\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:39.850473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.850387    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0f6996e-a749-41be-9a50-eaa334e1f584-ovnkube-script-lib\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.850540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.850483    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-daemon-config\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.850540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.850495    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93e7ee65-e872-4029-b310-a6f04d32c8fd-cni-binary-copy\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.850540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.850510    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.850691 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.850675    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/244d4caf-546f-46aa-bdf2-8023694c22a1-iptables-alerter-script\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:23:39.850726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.850698    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28838384-0663-4a33-9903-26afb7288d9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.851647 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.851632    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-original-pull-secret\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:39.858852 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.858832    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2mb\" (UniqueName: \"kubernetes.io/projected/9ced8467-be14-4a8a-a3ae-15566f579cc2-kube-api-access-vq2mb\") pod \"service-ca-operator-544d749c8c-6nbtd\" (UID: \"9ced8467-be14-4a8a-a3ae-15566f579cc2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd"
Nov 01 22:23:39.859001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.858986    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.859139 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.859123    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqct\" (UniqueName: \"kubernetes.io/projected/bb0df7fd-8362-4c2a-8258-e79901e6c402-kube-api-access-tvqct\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:23:39.859221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.859206    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kl5\" (UniqueName: \"kubernetes.io/projected/d66bf75c-5104-4918-b1f5-48fc89bf7f73-kube-api-access-c8kl5\") pod \"kas-connection-checker-685895c7c9-ktgdl\" (UID: \"d66bf75c-5104-4918-b1f5-48fc89bf7f73\") " pod="kube-system/kas-connection-checker-685895c7c9-ktgdl"
Nov 01 22:23:39.860288 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.860262    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxd8\" (UniqueName: \"kubernetes.io/projected/5b6492e2-6280-45e6-80df-537fa2489fdf-kube-api-access-xtxd8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb\" (UID: \"5b6492e2-6280-45e6-80df-537fa2489fdf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb"
Nov 01 22:23:39.861603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.861587    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgbc\" (UniqueName: \"kubernetes.io/projected/cbfec945-9ea9-4a51-a7fd-55714dfa01a6-kube-api-access-9sgbc\") pod \"ingress-canary-85ctr\" (UID: \"cbfec945-9ea9-4a51-a7fd-55714dfa01a6\") " pod="openshift-ingress-canary/ingress-canary-85ctr"
Nov 01 22:23:39.863600 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.863585    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9z9\" (UniqueName: \"kubernetes.io/projected/c9679225-9a70-433b-876a-6bd268ae64e4-kube-api-access-wl9z9\") pod \"insights-operator-7c4986d449-6n7x4\" (UID: \"c9679225-9a70-433b-876a-6bd268ae64e4\") " pod="openshift-insights/insights-operator-7c4986d449-6n7x4"
Nov 01 22:23:39.864094 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.864076    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfc7\" (UniqueName: \"kubernetes.io/projected/990ac23b-9dab-47c9-a8bf-13907483a733-kube-api-access-nhfc7\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:39.864265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.864240    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7rv\" (UniqueName: \"kubernetes.io/projected/ce89b463-4035-4c17-adc5-4a3e1c07e30b-kube-api-access-cv7rv\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:39.864498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.864478    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tds8r\" (UniqueName: \"kubernetes.io/projected/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-kube-api-access-tds8r\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:23:39.864580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.864478    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmv7\" (UniqueName: \"kubernetes.io/projected/cf0a7850-9ba3-4124-b0fe-a088bdfd05f2-kube-api-access-9xmv7\") pod \"volume-data-source-validator-58c9f5f46d-cwjn4\" (UID: \"cf0a7850-9ba3-4124-b0fe-a088bdfd05f2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwjn4"
Nov 01 22:23:39.864691 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.864673    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwbz\" (UniqueName: \"kubernetes.io/projected/93e7ee65-e872-4029-b310-a6f04d32c8fd-kube-api-access-ttwbz\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.864737 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.864690    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7r2\" (UniqueName: \"kubernetes.io/projected/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-api-access-rl7r2\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.865299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.865283    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6jq\" (UniqueName: \"kubernetes.io/projected/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kube-api-access-nn6jq\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.865523 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.865509    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x6dp\" (UniqueName: \"kubernetes.io/projected/b0f6996e-a749-41be-9a50-eaa334e1f584-kube-api-access-6x6dp\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.865781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.865767    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zn5\" (UniqueName: \"kubernetes.io/projected/234fcee4-4ad0-4144-8b1b-7f16e714a08f-kube-api-access-65zn5\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.866431 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.866410    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m46p\" (UniqueName: \"kubernetes.io/projected/28838384-0663-4a33-9903-26afb7288d9e-kube-api-access-6m46p\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.866559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.866543    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/a9038753-40e0-4e1f-aacc-da2a573be477-kube-api-access-mds7f\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.867059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.867027    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wjk\" (UniqueName: \"kubernetes.io/projected/244d4caf-546f-46aa-bdf2-8023694c22a1-kube-api-access-m7wjk\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:23:39.867117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.867055    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42kc\" (UniqueName: \"kubernetes.io/projected/dc731cae-588c-4d82-850e-1c3c3f954833-kube-api-access-l42kc\") pod \"kas-connection-checker-685895c7c9-rhtp4\" (UID: \"dc731cae-588c-4d82-850e-1c3c3f954833\") " pod="kube-system/kas-connection-checker-685895c7c9-rhtp4"
Nov 01 22:23:39.867187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.867173    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnkz\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.867287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.867273    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnp77\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.867612 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.867594    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxp7p\" (UniqueName: \"kubernetes.io/projected/6854396e-1bc7-4bab-8cd7-68d758d7d74f-kube-api-access-hxp7p\") pod \"network-check-target-j8gn5\" (UID: \"6854396e-1bc7-4bab-8cd7-68d758d7d74f\") " pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:39.867817 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.867803    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmv5\" (UniqueName: \"kubernetes.io/projected/1878f36c-168f-490a-bfef-bbe0f477bf88-kube-api-access-qjmv5\") pod \"kas-connection-checker-685895c7c9-gwk8z\" (UID: \"1878f36c-168f-490a-bfef-bbe0f477bf88\") " pod="kube-system/kas-connection-checker-685895c7c9-gwk8z"
Nov 01 22:23:39.868134 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.868108    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.868316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.868302    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t44kq\" (UniqueName: \"kubernetes.io/projected/d41ea86a-a166-44b7-a8da-2a7512c9e786-kube-api-access-t44kq\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:39.868512 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.868498    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rtr\" (UniqueName: \"kubernetes.io/projected/540ce336-6599-4411-bfc4-a475adfc2ecb-kube-api-access-86rtr\") pod \"network-check-source-6f78d59f99-4b4vw\" (UID: \"540ce336-6599-4411-bfc4-a475adfc2ecb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4b4vw"
Nov 01 22:23:39.868545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.868503    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r44ms\" (UniqueName: \"kubernetes.io/projected/2f63221e-2909-4490-bb43-cf2a6164d44c-kube-api-access-r44ms\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.869200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.869184    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmbj\" (UniqueName: \"kubernetes.io/projected/b330b73a-8364-478a-9589-1b24674fb30e-kube-api-access-qpmbj\") pod \"console-operator-7cd56b8dbc-m7tdl\" (UID: \"b330b73a-8364-478a-9589-1b24674fb30e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:39.886004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.885987    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:39.886107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886012    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-netd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.886107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886027    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-modprobe-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.886107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886064    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.886107 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886081    4837 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886114    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886121    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-netd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886082    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-ovn\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886143    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.386128449 +0000 UTC m=+1.975327487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886162    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-ovn\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886171    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-host\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886190    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886209    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.886247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886225    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886260    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-modprobe-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886277    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-host\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886297    4837 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886313    4837 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886332    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886358    4837 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886297    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886402    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.386391493 +0000 UTC m=+1.975590561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : secret "metrics-daemon-secret" not found
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886419    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886431    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.386413657 +0000 UTC m=+1.975612715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886456    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-socket-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886462    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-dbus\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886491    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-multus-certs\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886587    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886593    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-dbus\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886614    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-node-log\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886643    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-multus-certs\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886657    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-cnibin\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886675    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46fb3ebe-f172-40f8-ba72-3a2370cb447a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886682    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-run\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886691    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-node-log\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.886690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886703    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-lib-modules\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886722    4837 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886728    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-cnibin\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886738    4837 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886743    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-run\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886727    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-device-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886771    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-kubelet-config\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886776    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.386765045 +0000 UTC m=+1.975964083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886802    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-kubernetes\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886813    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e0e60c-b299-48da-86b6-3cf0dfdfa238-kubelet-config\") pod \"global-pull-secret-syncer-njrpj\" (UID: \"c3e0e60c-b299-48da-86b6-3cf0dfdfa238\") " pod="kube-system/global-pull-secret-syncer-njrpj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886823    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-bin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886815    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-device-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886828    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-lib-modules\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886842    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-kubernetes\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886848    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-bin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886846    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886889    4837 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.886926    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.386915375 +0000 UTC m=+1.976114444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886940    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886956    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886969    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-log-socket\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.886984    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46fb3ebe-f172-40f8-ba72-3a2370cb447a-crio-socket\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887000    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/244d4caf-546f-46aa-bdf2-8023694c22a1-host-slash\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887000    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887016    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-slash\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887035    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-log-socket\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887056    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46fb3ebe-f172-40f8-ba72-3a2370cb447a-crio-socket\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887061    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-slash\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887069    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/244d4caf-546f-46aa-bdf2-8023694c22a1-host-slash\") pod \"iptables-alerter-tb9n7\" (UID: \"244d4caf-546f-46aa-bdf2-8023694c22a1\") " pod="openshift-network-operator/iptables-alerter-tb9n7"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887082    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-var-lib-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887056    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-var-lib-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887069    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-sys-fs\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887111    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-hostroot\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887142    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887155    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-hostroot\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887162    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-etc-kubernetes\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887176    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-multus\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887192    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887207    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887222    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-kubelet\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887229    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-etc-kubernetes\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887239    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-conf-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887232    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-cni-multus\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887274    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-conf-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887286    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.887299    4837 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.887336    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.387327037 +0000 UTC m=+1.976526094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.887348    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.387342961 +0000 UTC m=+1.976542000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887346    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-kubelet\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887357    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887374    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-socket-dir-parent\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887392    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887419    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.887436    4837 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887444    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-systemd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887450    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.887462    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.387455619 +0000 UTC m=+1.976654662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887436    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-multus-socket-dir-parent\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887478    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-system-cni-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.887490    4837 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887495    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-run-systemd\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887501    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb0df7fd-8362-4c2a-8258-e79901e6c402-hosts-file\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.887528    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.387518552 +0000 UTC m=+1.976717618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887537    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb0df7fd-8362-4c2a-8258-e79901e6c402-hosts-file\") pod \"node-resolver-njpz8\" (UID: \"bb0df7fd-8362-4c2a-8258-e79901e6c402\") " pod="openshift-dns/node-resolver-njpz8"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887542    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-etc-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887549    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28838384-0663-4a33-9903-26afb7288d9e-system-cni-dir\") pod \"multus-additional-cni-plugins-zd476\" (UID: \"28838384-0663-4a33-9903-26afb7288d9e\") " pod="openshift-multus/multus-additional-cni-plugins-zd476"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887573    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-etc-openvswitch\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887572    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887588    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887599    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-var-lib-kubelet\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887615    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-k8s-cni-cncf-io\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887629    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-systemd-units\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887652    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-systemd-units\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887661    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-k8s-cni-cncf-io\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.887631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887664    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-var-lib-kubelet\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887686    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysconfig\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887707    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887732    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-netns\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887747    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-kubelet\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887748    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysconfig\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.887785    4837 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887811    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-var-lib-kubelet\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887814    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.887784    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-host-run-netns\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888093    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-system-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888138    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-netns\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888186    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888218    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:39.888232    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.388216718 +0000 UTC m=+1.977415762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888241    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-netns\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888289    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-system-cni-dir\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888298    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888380    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c026e341-04cc-4a50-bb3e-1ff663c2c7e9-registration-dir\") pod \"aws-ebs-csi-driver-node-qvvbb\" (UID: \"c026e341-04cc-4a50-bb3e-1ff663c2c7e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qvvbb"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888387    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-systemd\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888413    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-d\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888297    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-systemd\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888464    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-cnibin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888531    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-host\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888563    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-bin\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888526    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e7ee65-e872-4029-b310-a6f04d32c8fd-cnibin\") pod \"multus-r5xnp\" (UID: \"93e7ee65-e872-4029-b310-a6f04d32c8fd\") " pod="openshift-multus/multus-r5xnp"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888606    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-conf\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888635    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-sys\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888646    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0f6996e-a749-41be-9a50-eaa334e1f584-host-cni-bin\") pod \"ovnkube-node-n2phj\" (UID: \"b0f6996e-a749-41be-9a50-eaa334e1f584\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888588    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2b5233e-d5ac-4fa6-adff-97b78dcd1e40-host\") pod \"node-ca-km7mv\" (UID: \"e2b5233e-d5ac-4fa6-adff-97b78dcd1e40\") " pod="openshift-image-registry/node-ca-km7mv"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888776    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-sys\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.890746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.888942    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9038753-40e0-4e1f-aacc-da2a573be477-etc-sysctl-conf\") pod \"tuned-8g2k7\" (UID: \"a9038753-40e0-4e1f-aacc-da2a573be477\") " pod="openshift-cluster-node-tuning-operator/tuned-8g2k7"
Nov 01 22:23:39.936774 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.936729    4837 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnmh2_2ee57462-5ba3-4cfe-a233-13c046a7e797/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:39.971333 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:39.971318    4837 scope.go:122] "RemoveContainer" containerID="d6a6557aeab3afcd4e100f96b6cd67cffa9d11eb2f38f30d74e4cecd52aed739"
Nov 01 22:23:40.094741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.094429    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:23:40.392283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392223    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:40.392283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392265    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392288    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392337    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392354    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392360    4837 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392381    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392362    4837 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392409    4837 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392425    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392409104 +0000 UTC m=+2.981608143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392427    4837 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392404    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:40.392456 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392453    4837 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392462    4837 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392484    4837 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392485    4837 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392443    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.39243336 +0000 UTC m=+2.981632400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392468    4837 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392549    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392531907 +0000 UTC m=+2.981730950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392568    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392562627 +0000 UTC m=+2.981761665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392583    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392578338 +0000 UTC m=+2.981777376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : secret "metrics-daemon-secret" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392599    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392593701 +0000 UTC m=+2.981792740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392614    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392609117 +0000 UTC m=+2.981808156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392645    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392676    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.392696    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392759    4837 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392783    4837 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392797    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392778737 +0000 UTC m=+2.981977790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392819    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392808791 +0000 UTC m=+2.982007830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:40.392918 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.392837    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.392826706 +0000 UTC m=+2.982025748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:40.573146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.573125    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:23:40.573538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.573520    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/1.log"
Nov 01 22:23:40.573579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.573556    4837 generic.go:356] "Generic (PLEG): container finished" podID="b330b73a-8364-478a-9589-1b24674fb30e" containerID="875644fc4c1a6cea07710aedd4c6dc059cfa3e63c068cc277485f24809e54ca7" exitCode=255
Nov 01 22:23:40.573644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.573627    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" event={"ID":"b330b73a-8364-478a-9589-1b24674fb30e","Type":"ContainerDied","Data":"875644fc4c1a6cea07710aedd4c6dc059cfa3e63c068cc277485f24809e54ca7"}
Nov 01 22:23:40.573690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.573678    4837 scope.go:122] "RemoveContainer" containerID="d6a6557aeab3afcd4e100f96b6cd67cffa9d11eb2f38f30d74e4cecd52aed739"
Nov 01 22:23:40.574180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:40.574164    4837 scope.go:122] "RemoveContainer" containerID="875644fc4c1a6cea07710aedd4c6dc059cfa3e63c068cc277485f24809e54ca7"
Nov 01 22:23:40.574342 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:40.574323    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-m7tdl_openshift-console-operator(b330b73a-8364-478a-9589-1b24674fb30e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" podUID="b330b73a-8364-478a-9589-1b24674fb30e"
Nov 01 22:23:41.399963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.399930    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:41.400154 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.399971    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:41.400154 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.400008    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:41.400154 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.400065    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:41.400154 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400082    4837 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:41.400154 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.400096    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:41.400154 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400127    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.400115517 +0000 UTC m=+4.989314557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:41.400154 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400126    4837 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:41.400154 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.400144    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400132    4837 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400183    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.400174104 +0000 UTC m=+4.989373144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400188    4837 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400200    4837 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400214    4837 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400225    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.400208869 +0000 UTC m=+4.989407921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.400167    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400246    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.400236164 +0000 UTC m=+4.989435212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400218    4837 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400265    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.400256987 +0000 UTC m=+4.989456025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400290    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.40027645 +0000 UTC m=+4.989475502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : secret "metrics-daemon-secret" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400255    4837 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400310    4837 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400335    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.400328561 +0000 UTC m=+4.989527600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.400334    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.400375    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.400405    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:41.400479 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400463    4837 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:41.400974 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400500    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.400493846 +0000 UTC m=+4.989692915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:41.400974 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400526    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.40051695 +0000 UTC m=+4.989716005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:41.400974 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400558    4837 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:41.400974 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:41.400626    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.400614551 +0000 UTC m=+4.989813590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:41.578899 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.578875    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:23:41.631832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:41.631813    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:42.176697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:42.176673    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-njpz8_bb0df7fd-8362-4c2a-8258-e79901e6c402/dns-node-resolver/0.log"
Nov 01 22:23:43.176256 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.176231    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-km7mv_e2b5233e-d5ac-4fa6-adff-97b78dcd1e40/node-ca/0.log"
Nov 01 22:23:43.416853 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.416817    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:43.416853 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.416858    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.416880    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.416899    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.416941    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.416957    4837 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.416977    4837 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.416997    4837 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.416960    4837 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417016    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls podName:ce89b463-4035-4c17-adc5-4a3e1c07e30b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.416999926 +0000 UTC m=+9.006198982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vctzk" (UID: "ce89b463-4035-4c17-adc5-4a3e1c07e30b") : secret "samples-operator-tls" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417027    4837 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d5889b767-99spr: secret "image-registry-tls" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417034    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs podName:990ac23b-9dab-47c9-a8bf-13907483a733 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.417026862 +0000 UTC m=+9.006225905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs") pod "network-metrics-daemon-xvhpg" (UID: "990ac23b-9dab-47c9-a8bf-13907483a733") : secret "metrics-daemon-secret" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417038    4837 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.416975    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417016    4837 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677954f946-rrb8k: secret "image-registry-tls" not found
Nov 01 22:23:43.417087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417095    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls podName:46fb3ebe-f172-40f8-ba72-3a2370cb447a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.417080231 +0000 UTC m=+9.006279284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h84pf" (UID: "46fb3ebe-f172-40f8-ba72-3a2370cb447a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417103    4837 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417119    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls podName:5f9a27c9-af1d-47a8-96d1-e2618c5c0900 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.417106894 +0000 UTC m=+9.006305943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls") pod "image-registry-677954f946-rrb8k" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900") : secret "image-registry-tls" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417141    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls podName:f69ab461-32d6-44ee-912e-95c4c27d6696 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.417135946 +0000 UTC m=+9.006334985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls") pod "image-registry-d5889b767-99spr" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696") : secret "image-registry-tls" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417189    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.417173026 +0000 UTC m=+9.006372078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.417217    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.417259    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417277    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.417263728 +0000 UTC m=+9.006462788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.417305    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417330    4837 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.417344    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417363    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls podName:2f63221e-2909-4490-bb43-cf2a6164d44c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.41735546 +0000 UTC m=+9.006554502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls") pod "dns-default-ksjkx" (UID: "2f63221e-2909-4490-bb43-cf2a6164d44c") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417392    4837 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417425    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.417417169 +0000 UTC m=+9.006616227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417427    4837 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:43.417545 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:43.417451    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls podName:d41ea86a-a166-44b7-a8da-2a7512c9e786 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.417445207 +0000 UTC m=+9.006644276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ss6wx" (UID: "d41ea86a-a166-44b7-a8da-2a7512c9e786") : secret "cluster-monitoring-operator-tls" not found
Nov 01 22:23:43.782343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:43.782303    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-85ctr_cbfec945-9ea9-4a51-a7fd-55714dfa01a6/serve-healthcheck-canary/0.log"
Nov 01 22:23:44.220515 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:44.220473    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:44.220875 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:44.220800    4837 scope.go:122] "RemoveContainer" containerID="875644fc4c1a6cea07710aedd4c6dc059cfa3e63c068cc277485f24809e54ca7"
Nov 01 22:23:44.220985 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:44.220967    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-m7tdl_openshift-console-operator(b330b73a-8364-478a-9589-1b24674fb30e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" podUID="b330b73a-8364-478a-9589-1b24674fb30e"
Nov 01 22:23:44.779435 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:44.779408    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb_5b6492e2-6280-45e6-80df-537fa2489fdf/kube-storage-version-migrator-operator/0.log"
Nov 01 22:23:46.833635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:46.833605    4837 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:46.834018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:46.833940    4837 scope.go:122] "RemoveContainer" containerID="875644fc4c1a6cea07710aedd4c6dc059cfa3e63c068cc277485f24809e54ca7"
Nov 01 22:23:46.834134 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:46.834114    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-m7tdl_openshift-console-operator(b330b73a-8364-478a-9589-1b24674fb30e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" podUID="b330b73a-8364-478a-9589-1b24674fb30e"
Nov 01 22:23:47.456604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456572    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:47.456814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456613    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:47.456814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456634    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:47.456814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456654    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:47.456814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456695    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:47.456814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456729    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:47.456814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456757    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:47.456814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456800    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:47.457230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456827    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:47.457230 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:47.456857    4837 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 22:23:47.457230 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:47.456934    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:55.456913974 +0000 UTC m=+17.046113018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : secret "router-metrics-certs-default" not found
Nov 01 22:23:47.457230 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:47.456934    4837 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 22:23:47.457230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456865    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:47.457230 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:47.456978    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert podName:9a69bb5e-ee33-44a8-912b-cc5e3d017d7b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:55.456967825 +0000 UTC m=+17.046166867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ths57" (UID: "9a69bb5e-ee33-44a8-912b-cc5e3d017d7b") : secret "networking-console-plugin-cert" not found
Nov 01 22:23:47.457230 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:23:47.457001    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle podName:234fcee4-4ad0-4144-8b1b-7f16e714a08f nodeName:}" failed. No retries permitted until 2026-11-01 22:23:55.456992403 +0000 UTC m=+17.046191450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle") pod "router-default-56cdb546f-44l9r" (UID: "234fcee4-4ad0-4144-8b1b-7f16e714a08f") : configmap references non-existent config key: service-ca.crt
Nov 01 22:23:47.457230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.456983    4837 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 22:23:47.459499 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.459472    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"image-registry-d5889b767-99spr\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") " pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:47.459698 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.459675    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46fb3ebe-f172-40f8-ba72-3a2370cb447a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h84pf\" (UID: \"46fb3ebe-f172-40f8-ba72-3a2370cb447a\") " pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:47.459850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.459824    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d41ea86a-a166-44b7-a8da-2a7512c9e786-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ss6wx\" (UID: \"d41ea86a-a166-44b7-a8da-2a7512c9e786\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:47.459945 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.459868    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f63221e-2909-4490-bb43-cf2a6164d44c-metrics-tls\") pod \"dns-default-ksjkx\" (UID: \"2f63221e-2909-4490-bb43-cf2a6164d44c\") " pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:47.459945 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.459884    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/990ac23b-9dab-47c9-a8bf-13907483a733-metrics-certs\") pod \"network-metrics-daemon-xvhpg\" (UID: \"990ac23b-9dab-47c9-a8bf-13907483a733\") " pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:47.460082 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.460035    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ce89b463-4035-4c17-adc5-4a3e1c07e30b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vctzk\" (UID: \"ce89b463-4035-4c17-adc5-4a3e1c07e30b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:47.460213 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.460197    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"image-registry-677954f946-rrb8k\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") " pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:47.470988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.470969    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:47.471101 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.471067    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"
Nov 01 22:23:47.472061 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.472032    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"
Nov 01 22:23:47.476387 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.476365    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:47.476458 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.476424    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvhpg"
Nov 01 22:23:47.480498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.480476    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:47.481563 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.481545    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h84pf"
Nov 01 22:23:47.943236 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.943214    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk"]
Nov 01 22:23:47.981624 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.981604    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:23:47.982499 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:47.982474    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990ac23b_9dab_47c9_a8bf_13907483a733.slice/crio-a7dcc055da9feb33afc8563b167977e3d089b349f8c59e09c3a50df83e340ca5 WatchSource:0}: Error finding container a7dcc055da9feb33afc8563b167977e3d089b349f8c59e09c3a50df83e340ca5: Status 404 returned error can't find the container with id a7dcc055da9feb33afc8563b167977e3d089b349f8c59e09c3a50df83e340ca5
Nov 01 22:23:47.984549 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:47.984520    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41ea86a_a166_44b7_a8da_2a7512c9e786.slice/crio-999b7efd563d9e7ab73bf4191cd0fdd6c54dc1ac50bbf85b7702d7c9e4ea567e WatchSource:0}: Error finding container 999b7efd563d9e7ab73bf4191cd0fdd6c54dc1ac50bbf85b7702d7c9e4ea567e: Status 404 returned error can't find the container with id 999b7efd563d9e7ab73bf4191cd0fdd6c54dc1ac50bbf85b7702d7c9e4ea567e
Nov 01 22:23:47.985308 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:47.985285    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9a27c9_af1d_47a8_96d1_e2618c5c0900.slice/crio-081043a60f3d9d7d771f376d2930c3cc4a5268644325a931faeac0a5ab2d7523 WatchSource:0}: Error finding container 081043a60f3d9d7d771f376d2930c3cc4a5268644325a931faeac0a5ab2d7523: Status 404 returned error can't find the container with id 081043a60f3d9d7d771f376d2930c3cc4a5268644325a931faeac0a5ab2d7523
Nov 01 22:23:47.990102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.990084    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xvhpg"]
Nov 01 22:23:47.990629 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.990612    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx"]
Nov 01 22:23:47.993284 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:47.993264    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69ab461_32d6_44ee_912e_95c4c27d6696.slice/crio-13d74dbbda00d00886d42fb3d68813c7c0cfa0cf50a85a61d7ad5b39b565c2ba WatchSource:0}: Error finding container 13d74dbbda00d00886d42fb3d68813c7c0cfa0cf50a85a61d7ad5b39b565c2ba: Status 404 returned error can't find the container with id 13d74dbbda00d00886d42fb3d68813c7c0cfa0cf50a85a61d7ad5b39b565c2ba
Nov 01 22:23:47.993375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:47.993358    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-677954f946-rrb8k"]
Nov 01 22:23:47.993815 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:47.993790    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f63221e_2909_4490_bb43_cf2a6164d44c.slice/crio-0e67276c056972e41ef6357362fc075bd98c3e3b2ef485922f63975025855fc5 WatchSource:0}: Error finding container 0e67276c056972e41ef6357362fc075bd98c3e3b2ef485922f63975025855fc5: Status 404 returned error can't find the container with id 0e67276c056972e41ef6357362fc075bd98c3e3b2ef485922f63975025855fc5
Nov 01 22:23:47.998291 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:47.998169    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fb3ebe_f172_40f8_ba72_3a2370cb447a.slice/crio-85020aba885da7e505c4acfd0e95461fb0562710dbf07810a923d526a8cb6398 WatchSource:0}: Error finding container 85020aba885da7e505c4acfd0e95461fb0562710dbf07810a923d526a8cb6398: Status 404 returned error can't find the container with id 85020aba885da7e505c4acfd0e95461fb0562710dbf07810a923d526a8cb6398
Nov 01 22:23:48.036654 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.036631    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d5889b767-99spr"]
Nov 01 22:23:48.037374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.037355    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ksjkx"]
Nov 01 22:23:48.037703 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.037685    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h84pf"]
Nov 01 22:23:48.265931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.265846    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:48.266153 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.266138    4837 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 22:23:48.608877 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.608794    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h84pf" event={"ID":"46fb3ebe-f172-40f8-ba72-3a2370cb447a","Type":"ContainerStarted","Data":"ed6c77053037bc04abbd1e9da3c2360458d1532ab30c5fe93e6ae48e3cb27fe4"}
Nov 01 22:23:48.609006 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.608880    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h84pf" event={"ID":"46fb3ebe-f172-40f8-ba72-3a2370cb447a","Type":"ContainerStarted","Data":"85020aba885da7e505c4acfd0e95461fb0562710dbf07810a923d526a8cb6398"}
Nov 01 22:23:48.609936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.609914    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xvhpg" event={"ID":"990ac23b-9dab-47c9-a8bf-13907483a733","Type":"ContainerStarted","Data":"a7dcc055da9feb33afc8563b167977e3d089b349f8c59e09c3a50df83e340ca5"}
Nov 01 22:23:48.611008 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.610985    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk" event={"ID":"ce89b463-4035-4c17-adc5-4a3e1c07e30b","Type":"ContainerStarted","Data":"654512d55f1e8212d1c75ace3aff2eddd857e1668b4a22db8d264190fd2620df"}
Nov 01 22:23:48.612341 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.612317    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5889b767-99spr" event={"ID":"f69ab461-32d6-44ee-912e-95c4c27d6696","Type":"ContainerStarted","Data":"811de0daf11b6f8eda0911701e789563b25ba8325e7ec05d7b00ec16c1996805"}
Nov 01 22:23:48.612431 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.612355    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5889b767-99spr" event={"ID":"f69ab461-32d6-44ee-912e-95c4c27d6696","Type":"ContainerStarted","Data":"13d74dbbda00d00886d42fb3d68813c7c0cfa0cf50a85a61d7ad5b39b565c2ba"}
Nov 01 22:23:48.613455 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.613436    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ksjkx" event={"ID":"2f63221e-2909-4490-bb43-cf2a6164d44c","Type":"ContainerStarted","Data":"0e67276c056972e41ef6357362fc075bd98c3e3b2ef485922f63975025855fc5"}
Nov 01 22:23:48.614526 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.614508    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx" event={"ID":"d41ea86a-a166-44b7-a8da-2a7512c9e786","Type":"ContainerStarted","Data":"999b7efd563d9e7ab73bf4191cd0fdd6c54dc1ac50bbf85b7702d7c9e4ea567e"}
Nov 01 22:23:48.615787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.615766    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-677954f946-rrb8k" event={"ID":"5f9a27c9-af1d-47a8-96d1-e2618c5c0900","Type":"ContainerStarted","Data":"faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71"}
Nov 01 22:23:48.615879 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.615801    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-677954f946-rrb8k" event={"ID":"5f9a27c9-af1d-47a8-96d1-e2618c5c0900","Type":"ContainerStarted","Data":"081043a60f3d9d7d771f376d2930c3cc4a5268644325a931faeac0a5ab2d7523"}
Nov 01 22:23:48.615940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:48.615916    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:23:49.623638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:49.623560    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h84pf" event={"ID":"46fb3ebe-f172-40f8-ba72-3a2370cb447a","Type":"ContainerStarted","Data":"46fccbd9085f8fdad6c290597439ba1b568024cacc9b7cb4e2f1c1f4ea19dcde"}
Nov 01 22:23:49.624231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:49.623664    4837 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 22:23:50.630286 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:50.630239    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xvhpg" event={"ID":"990ac23b-9dab-47c9-a8bf-13907483a733","Type":"ContainerStarted","Data":"b32d4e08813045bfedc56c2974da8f4a67cb24fcb8c4a368267a9422cf1ee58e"}
Nov 01 22:23:50.630772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:50.630315    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xvhpg" event={"ID":"990ac23b-9dab-47c9-a8bf-13907483a733","Type":"ContainerStarted","Data":"e6cabbd9795d64e8aaf867d9c7724efdbe4963def933b478549c7a594bcbdc58"}
Nov 01 22:23:50.955302 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:50.955276    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:23:51.517440 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.517411    4837 scope.go:122] "RemoveContainer" containerID="875644fc4c1a6cea07710aedd4c6dc059cfa3e63c068cc277485f24809e54ca7"
Nov 01 22:23:51.638930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.638894    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:23:51.639374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.639011    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" event={"ID":"b330b73a-8364-478a-9589-1b24674fb30e","Type":"ContainerStarted","Data":"ca436c2eabe6058feb4931a01e2cb62beedf8b4a137a4d87084c7fa8f8ce9bc9"}
Nov 01 22:23:51.639444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.639415    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:51.640290 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.640263    4837 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-m7tdl container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.20:8443/readyz\": dial tcp 10.134.0.20:8443: connect: connection refused" start-of-body=
Nov 01 22:23:51.640410 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.640311    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl" podUID="b330b73a-8364-478a-9589-1b24674fb30e" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.20:8443/readyz\": dial tcp 10.134.0.20:8443: connect: connection refused"
Nov 01 22:23:51.641148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.641117    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk" event={"ID":"ce89b463-4035-4c17-adc5-4a3e1c07e30b","Type":"ContainerStarted","Data":"43f7652b53b7322bc8a93871bfac5fb8886abb6e37096352f0f0311049c028cd"}
Nov 01 22:23:51.641250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.641166    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vctzk" event={"ID":"ce89b463-4035-4c17-adc5-4a3e1c07e30b","Type":"ContainerStarted","Data":"78438e67afcb0df6edae0e8fb0506dc3944a1ae71ea0047ef89bbe14522b12fb"}
Nov 01 22:23:51.643393 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.643365    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ksjkx" event={"ID":"2f63221e-2909-4490-bb43-cf2a6164d44c","Type":"ContainerStarted","Data":"ea7889672db0e5a845fa68cf8b2d83227b6df49e96c9e44301fd2e32dfc7aa73"}
Nov 01 22:23:51.643487 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.643399    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ksjkx" event={"ID":"2f63221e-2909-4490-bb43-cf2a6164d44c","Type":"ContainerStarted","Data":"9e730df00fe2a55790ec5a33de33384f4d4a4a1cff6c31a0dd1408ff43dc18c9"}
Nov 01 22:23:51.643487 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.643442    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:23:51.645596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:51.645545    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ss6wx" event={"ID":"d41ea86a-a166-44b7-a8da-2a7512c9e786","Type":"ContainerStarted","Data":"8158de3eb535af967409dbc48c95b193c6004644600d940c15395e9b8cf012eb"}
Nov 01 22:23:52.010230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:52.010201    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2phj"
Nov 01 22:23:52.654742 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:52.654703    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h84pf" event={"ID":"46fb3ebe-f172-40f8-ba72-3a2370cb447a","Type":"ContainerStarted","Data":"d46d927c439becc3ae449c659af8a4cd627084a82ade98643aaf12485e37a7e1"}
Nov 01 22:23:52.659346 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:52.659324    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-m7tdl"
Nov 01 22:23:53.883536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:53.883510    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hnmh2"
Nov 01 22:23:54.857184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:54.857154    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-677954f946-rrb8k"]
Nov 01 22:23:55.537673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:55.537636    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:55.538033 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:55.537724    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:55.538033 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:55.537761    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:55.538484 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:55.538460    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/234fcee4-4ad0-4144-8b1b-7f16e714a08f-service-ca-bundle\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:55.540183 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:55.540162    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a69bb5e-ee33-44a8-912b-cc5e3d017d7b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ths57\" (UID: \"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:55.540234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:55.540224    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234fcee4-4ad0-4144-8b1b-7f16e714a08f-metrics-certs\") pod \"router-default-56cdb546f-44l9r\" (UID: \"234fcee4-4ad0-4144-8b1b-7f16e714a08f\") " pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:55.573098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:55.573070    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:55.573213 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:55.573125    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57"
Nov 01 22:23:56.032683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:56.032638    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ths57"]
Nov 01 22:23:56.034806 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:56.034772    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a69bb5e_ee33_44a8_912b_cc5e3d017d7b.slice/crio-8729182c540e439a65f1dddb868e60852b1a994927e3734189dc046c4b64bff6 WatchSource:0}: Error finding container 8729182c540e439a65f1dddb868e60852b1a994927e3734189dc046c4b64bff6: Status 404 returned error can't find the container with id 8729182c540e439a65f1dddb868e60852b1a994927e3734189dc046c4b64bff6
Nov 01 22:23:56.035381 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:56.035360    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56cdb546f-44l9r"]
Nov 01 22:23:56.080128 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:23:56.080105    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234fcee4_4ad0_4144_8b1b_7f16e714a08f.slice/crio-1b3273e94b1eb514996f330f4f0adf67c75844eb5290c8b22845dac988646b0e WatchSource:0}: Error finding container 1b3273e94b1eb514996f330f4f0adf67c75844eb5290c8b22845dac988646b0e: Status 404 returned error can't find the container with id 1b3273e94b1eb514996f330f4f0adf67c75844eb5290c8b22845dac988646b0e
Nov 01 22:23:56.670337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:56.670300    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56cdb546f-44l9r" event={"ID":"234fcee4-4ad0-4144-8b1b-7f16e714a08f","Type":"ContainerStarted","Data":"111be47900208e48a8a34b914694a408582695d72a714dbb81fcf0c0dbe12648"}
Nov 01 22:23:56.670770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:56.670361    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56cdb546f-44l9r" event={"ID":"234fcee4-4ad0-4144-8b1b-7f16e714a08f","Type":"ContainerStarted","Data":"1b3273e94b1eb514996f330f4f0adf67c75844eb5290c8b22845dac988646b0e"}
Nov 01 22:23:56.671568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:56.671542    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57" event={"ID":"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b","Type":"ContainerStarted","Data":"8729182c540e439a65f1dddb868e60852b1a994927e3734189dc046c4b64bff6"}
Nov 01 22:23:57.573298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:57.573271    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:57.576181 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:57.576161    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:57.675989 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:57.675958    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-ths57" event={"ID":"9a69bb5e-ee33-44a8-912b-cc5e3d017d7b","Type":"ContainerStarted","Data":"2f4e76613a66b0e218adba84032748f7ed86e4395db9dafee9855f2ab26ce2b9"}
Nov 01 22:23:57.676321 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:57.676272    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:23:57.677273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:23:57.677253    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56cdb546f-44l9r"
Nov 01 22:24:00.072689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:00.072651    4837 patch_prober.go:29] interesting pod/image-registry-d5889b767-99spr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 22:24:00.073155 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:00.072722    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d5889b767-99spr" podUID="f69ab461-32d6-44ee-912e-95c4c27d6696" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:24:01.657652 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:01.657624    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ksjkx"
Nov 01 22:24:04.861709 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:04.861679    4837 patch_prober.go:29] interesting pod/image-registry-677954f946-rrb8k container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 22:24:04.862089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:04.861725    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-677954f946-rrb8k" podUID="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:24:05.531013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.530975    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q6djl"]
Nov 01 22:24:05.562602 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.562568    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.638514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.638485    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 22:24:05.657992 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.657965    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 22:24:05.658119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.658027    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8bwdj\""
Nov 01 22:24:05.700105 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.700072    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 22:24:05.704740 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.704719    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 22:24:05.715021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715001    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d16f06c-40ec-43c7-8a0a-8959bb077e10-metrics-client-ca\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.715126 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715034    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2jm\" (UniqueName: \"kubernetes.io/projected/5d16f06c-40ec-43c7-8a0a-8959bb077e10-kube-api-access-br2jm\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.715126 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715085    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.715208 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715155    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-sys\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.715208 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715175    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.715208 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715203    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-textfile\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.715307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715280    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-root\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.715348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715312    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-tls\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.715404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.715390    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-wtmp\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.815846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.815774    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d16f06c-40ec-43c7-8a0a-8959bb077e10-metrics-client-ca\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.815846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.815819    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2jm\" (UniqueName: \"kubernetes.io/projected/5d16f06c-40ec-43c7-8a0a-8959bb077e10-kube-api-access-br2jm\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.815846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.815842    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.816098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.816011    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-sys\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.816098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.816082    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.816202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.816126    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-sys\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.816252 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.816201    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-textfile\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.816252 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.816239    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-root\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.816354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.816266    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-tls\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.816354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.816317    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-wtmp\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.817119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.817089    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.817297 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.817231    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-wtmp\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.818679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.817839    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-textfile\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.818679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.818073    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d16f06c-40ec-43c7-8a0a-8959bb077e10-metrics-client-ca\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.820978 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.820952    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.822071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.822032    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d16f06c-40ec-43c7-8a0a-8959bb077e10-node-exporter-tls\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.825882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.825859    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d16f06c-40ec-43c7-8a0a-8959bb077e10-root\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.868997 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.868970    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2jm\" (UniqueName: \"kubernetes.io/projected/5d16f06c-40ec-43c7-8a0a-8959bb077e10-kube-api-access-br2jm\") pod \"node-exporter-q6djl\" (UID: \"5d16f06c-40ec-43c7-8a0a-8959bb077e10\") " pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.914930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:05.914902    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q6djl"
Nov 01 22:24:05.948292 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:24:05.948260    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d16f06c_40ec_43c7_8a0a_8959bb077e10.slice/crio-4c205929072e9e492e1e67ed482a8960a2bc641a14afcf1c100a2e29c8cb38f8 WatchSource:0}: Error finding container 4c205929072e9e492e1e67ed482a8960a2bc641a14afcf1c100a2e29c8cb38f8: Status 404 returned error can't find the container with id 4c205929072e9e492e1e67ed482a8960a2bc641a14afcf1c100a2e29c8cb38f8
Nov 01 22:24:06.707286 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:06.707253    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6djl" event={"ID":"5d16f06c-40ec-43c7-8a0a-8959bb077e10","Type":"ContainerStarted","Data":"4c205929072e9e492e1e67ed482a8960a2bc641a14afcf1c100a2e29c8cb38f8"}
Nov 01 22:24:08.716897 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:08.716863    4837 generic.go:356] "Generic (PLEG): container finished" podID="5d16f06c-40ec-43c7-8a0a-8959bb077e10" containerID="2a382912ef9d8e2bf4c7ecfb798b3bc07ffbacc515d8ec6d7b236f71495c74cd" exitCode=0
Nov 01 22:24:08.716897 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:08.716901    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6djl" event={"ID":"5d16f06c-40ec-43c7-8a0a-8959bb077e10","Type":"ContainerDied","Data":"2a382912ef9d8e2bf4c7ecfb798b3bc07ffbacc515d8ec6d7b236f71495c74cd"}
Nov 01 22:24:09.722639 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:09.722603    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6djl" event={"ID":"5d16f06c-40ec-43c7-8a0a-8959bb077e10","Type":"ContainerStarted","Data":"9a0bbd5ef83be3fb874ad34f0ff5203b34b5a8f50da488193c7b637880f2eaae"}
Nov 01 22:24:09.723122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:09.722697    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6djl" event={"ID":"5d16f06c-40ec-43c7-8a0a-8959bb077e10","Type":"ContainerStarted","Data":"cfbff827c90d09e8ce16b7462d56448f628947afaebee1f2ceb6485e3a4f9efc"}
Nov 01 22:24:10.576692 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:10.576660    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j8gn5"
Nov 01 22:24:10.682646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:10.682599    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q6djl" podStartSLOduration=3.557608536 podStartE2EDuration="5.682587219s" totalImagesPullingTime="2.047280234s" totalInitContainerRuntime="77.698449ms" isStatefulPod=true podCreationTimestamp="2026-11-01 22:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:09.831273706 +0000 UTC m=+31.420472766" watchObservedRunningTime="2026-11-01 22:24:10.682587219 +0000 UTC m=+32.271786279"
Nov 01 22:24:10.960422 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:10.960394    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:24:14.862110 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:14.862084    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:24:17.589739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:17.589707    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d5889b767-99spr"]
Nov 01 22:24:18.811572 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:24:18.811539    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:24:19.876427 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:19.876388    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-677954f946-rrb8k" podUID="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" containerName="registry" containerID="cri-o://faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71" gracePeriod=30
Nov 01 22:24:20.117378 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.117352    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:24:20.223449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223414    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca\") pod \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") "
Nov 01 22:24:20.223449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223455    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token\") pod \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") "
Nov 01 22:24:20.223694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223495    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates\") pod \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") "
Nov 01 22:24:20.223694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223519    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77\") pod \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") "
Nov 01 22:24:20.223694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223565    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets\") pod \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") "
Nov 01 22:24:20.223694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223591    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted\") pod \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") "
Nov 01 22:24:20.223694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223630    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") pod \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") "
Nov 01 22:24:20.223694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223665    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration\") pod \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\" (UID: \"5f9a27c9-af1d-47a8-96d1-e2618c5c0900\") "
Nov 01 22:24:20.224021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.223977    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca" pod "5f9a27c9-af1d-47a8-96d1-e2618c5c0900" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:20.224210 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.224186    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates" pod "5f9a27c9-af1d-47a8-96d1-e2618c5c0900" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:20.226159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.226099    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls" pod "5f9a27c9-af1d-47a8-96d1-e2618c5c0900" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:20.226298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.226263    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets" pod "5f9a27c9-af1d-47a8-96d1-e2618c5c0900" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:24:20.226484 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.226453    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration" pod "5f9a27c9-af1d-47a8-96d1-e2618c5c0900" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:24:20.226559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.226488    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token" pod "5f9a27c9-af1d-47a8-96d1-e2618c5c0900" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:20.226604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.226573    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77" pod "5f9a27c9-af1d-47a8-96d1-e2618c5c0900" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900"). InnerVolumeSpecName "kube-api-access-hnp77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:20.231037 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.230922    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted" pod "5f9a27c9-af1d-47a8-96d1-e2618c5c0900" (UID: "5f9a27c9-af1d-47a8-96d1-e2618c5c0900"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:24:20.324937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.324908    4837 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-trusted-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:20.324937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.324932    4837 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-bound-sa-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:20.324937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.324943    4837 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-certificates\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:20.325150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.324953    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnp77\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-kube-api-access-hnp77\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:20.325150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.324962    4837 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-installation-pull-secrets\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:20.325150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.324971    4837 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-ca-trust-extracted\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:20.325150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.324979    4837 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-registry-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:20.325150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.324989    4837 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9a27c9-af1d-47a8-96d1-e2618c5c0900-image-registry-private-configuration\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:20.764389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.764354    4837 generic.go:356] "Generic (PLEG): container finished" podID="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" containerID="faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71" exitCode=0
Nov 01 22:24:20.764573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.764411    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677954f946-rrb8k"
Nov 01 22:24:20.764573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.764427    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-677954f946-rrb8k" event={"ID":"5f9a27c9-af1d-47a8-96d1-e2618c5c0900","Type":"ContainerDied","Data":"faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71"}
Nov 01 22:24:20.764573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.764467    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-677954f946-rrb8k" event={"ID":"5f9a27c9-af1d-47a8-96d1-e2618c5c0900","Type":"ContainerDied","Data":"081043a60f3d9d7d771f376d2930c3cc4a5268644325a931faeac0a5ab2d7523"}
Nov 01 22:24:20.764573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.764485    4837 scope.go:122] "RemoveContainer" containerID="faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71"
Nov 01 22:24:20.772534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.772508    4837 scope.go:122] "RemoveContainer" containerID="faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71"
Nov 01 22:24:20.772859 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:24:20.772829    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71\": container with ID starting with faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71 not found: ID does not exist" containerID="faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71"
Nov 01 22:24:20.772932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.772866    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71"} err="failed to get container status \"faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71\": rpc error: code = NotFound desc = could not find container \"faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71\": container with ID starting with faa69010167a41d7e292a08bb6dbfc9e4a9c2f909741920be90c979fb698ac71 not found: ID does not exist"
Nov 01 22:24:20.978109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:20.978079    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-677954f946-rrb8k"]
Nov 01 22:24:21.024921 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:21.024895    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-677954f946-rrb8k"]
Nov 01 22:24:22.523437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:22.523404    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" path="/var/lib/kubelet/pods/5f9a27c9-af1d-47a8-96d1-e2618c5c0900/volumes"
Nov 01 22:24:42.608772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.608716    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d5889b767-99spr" podUID="f69ab461-32d6-44ee-912e-95c4c27d6696" containerName="registry" containerID="cri-o://811de0daf11b6f8eda0911701e789563b25ba8325e7ec05d7b00ec16c1996805" gracePeriod=30
Nov 01 22:24:42.855291 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.855261    4837 generic.go:356] "Generic (PLEG): container finished" podID="f69ab461-32d6-44ee-912e-95c4c27d6696" containerID="811de0daf11b6f8eda0911701e789563b25ba8325e7ec05d7b00ec16c1996805" exitCode=0
Nov 01 22:24:42.855421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.855336    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5889b767-99spr" event={"ID":"f69ab461-32d6-44ee-912e-95c4c27d6696","Type":"ContainerDied","Data":"811de0daf11b6f8eda0911701e789563b25ba8325e7ec05d7b00ec16c1996805"}
Nov 01 22:24:42.871218 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.871172    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:24:42.978704 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.978675    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted\") pod \"f69ab461-32d6-44ee-912e-95c4c27d6696\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") "
Nov 01 22:24:42.978857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.978717    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets\") pod \"f69ab461-32d6-44ee-912e-95c4c27d6696\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") "
Nov 01 22:24:42.978857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.978738    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz\") pod \"f69ab461-32d6-44ee-912e-95c4c27d6696\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") "
Nov 01 22:24:42.978857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.978760    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token\") pod \"f69ab461-32d6-44ee-912e-95c4c27d6696\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") "
Nov 01 22:24:42.978857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.978778    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates\") pod \"f69ab461-32d6-44ee-912e-95c4c27d6696\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") "
Nov 01 22:24:42.978857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.978800    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") pod \"f69ab461-32d6-44ee-912e-95c4c27d6696\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") "
Nov 01 22:24:42.978857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.978834    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration\") pod \"f69ab461-32d6-44ee-912e-95c4c27d6696\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") "
Nov 01 22:24:42.979202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.978911    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca\") pod \"f69ab461-32d6-44ee-912e-95c4c27d6696\" (UID: \"f69ab461-32d6-44ee-912e-95c4c27d6696\") "
Nov 01 22:24:42.979538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.979369    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates" pod "f69ab461-32d6-44ee-912e-95c4c27d6696" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:42.979660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.979616    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca" pod "f69ab461-32d6-44ee-912e-95c4c27d6696" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:42.981409 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.981379    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration" pod "f69ab461-32d6-44ee-912e-95c4c27d6696" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:24:42.981504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.981417    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token" pod "f69ab461-32d6-44ee-912e-95c4c27d6696" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:42.981504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.981448    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls" pod "f69ab461-32d6-44ee-912e-95c4c27d6696" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:42.981594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.981502    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets" pod "f69ab461-32d6-44ee-912e-95c4c27d6696" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:24:42.981631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.981583    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz" pod "f69ab461-32d6-44ee-912e-95c4c27d6696" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696"). InnerVolumeSpecName "kube-api-access-9rnkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:42.985817 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:42.985800    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted" pod "f69ab461-32d6-44ee-912e-95c4c27d6696" (UID: "f69ab461-32d6-44ee-912e-95c4c27d6696"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:24:43.079923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.079896    4837 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-installation-pull-secrets\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:43.079923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.079920    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rnkz\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-kube-api-access-9rnkz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:43.080102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.079937    4837 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-bound-sa-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:43.080102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.079950    4837 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-certificates\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:43.080102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.079958    4837 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f69ab461-32d6-44ee-912e-95c4c27d6696-registry-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:43.080102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.079967    4837 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f69ab461-32d6-44ee-912e-95c4c27d6696-image-registry-private-configuration\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:43.080102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.079976    4837 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f69ab461-32d6-44ee-912e-95c4c27d6696-trusted-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:43.080102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.079988    4837 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f69ab461-32d6-44ee-912e-95c4c27d6696-ca-trust-extracted\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:43.860200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.860172    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5889b767-99spr"
Nov 01 22:24:43.860592 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.860173    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5889b767-99spr" event={"ID":"f69ab461-32d6-44ee-912e-95c4c27d6696","Type":"ContainerDied","Data":"13d74dbbda00d00886d42fb3d68813c7c0cfa0cf50a85a61d7ad5b39b565c2ba"}
Nov 01 22:24:43.860592 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:43.860302    4837 scope.go:122] "RemoveContainer" containerID="811de0daf11b6f8eda0911701e789563b25ba8325e7ec05d7b00ec16c1996805"
Nov 01 22:24:44.060588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:44.060556    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d5889b767-99spr"]
Nov 01 22:24:44.107341 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:44.107316    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d5889b767-99spr"]
Nov 01 22:24:44.525005 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:44.524972    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69ab461-32d6-44ee-912e-95c4c27d6696" path="/var/lib/kubelet/pods/f69ab461-32d6-44ee-912e-95c4c27d6696/volumes"
Nov 01 22:24:48.810283 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:24:48.810252    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:24:52.898477 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:52.898445    4837 generic.go:356] "Generic (PLEG): container finished" podID="c9679225-9a70-433b-876a-6bd268ae64e4" containerID="cb5b9d48495c813b0abcc7094d7b08715e65df7c6f487c01d4852dc8d4289569" exitCode=0
Nov 01 22:24:52.898807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:52.898520    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6n7x4" event={"ID":"c9679225-9a70-433b-876a-6bd268ae64e4","Type":"ContainerDied","Data":"cb5b9d48495c813b0abcc7094d7b08715e65df7c6f487c01d4852dc8d4289569"}
Nov 01 22:24:52.898847 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:52.898808    4837 scope.go:122] "RemoveContainer" containerID="cb5b9d48495c813b0abcc7094d7b08715e65df7c6f487c01d4852dc8d4289569"
Nov 01 22:24:53.903152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:53.903118    4837 generic.go:356] "Generic (PLEG): container finished" podID="9ced8467-be14-4a8a-a3ae-15566f579cc2" containerID="00c0d4d3492693cc8b67f9fbd8cc146964a8e54f9c170a5ffe9b2caf1585c428" exitCode=0
Nov 01 22:24:53.903531 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:53.903191    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd" event={"ID":"9ced8467-be14-4a8a-a3ae-15566f579cc2","Type":"ContainerDied","Data":"00c0d4d3492693cc8b67f9fbd8cc146964a8e54f9c170a5ffe9b2caf1585c428"}
Nov 01 22:24:53.903591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:53.903532    4837 scope.go:122] "RemoveContainer" containerID="00c0d4d3492693cc8b67f9fbd8cc146964a8e54f9c170a5ffe9b2caf1585c428"
Nov 01 22:24:53.905062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:53.905030    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-6n7x4" event={"ID":"c9679225-9a70-433b-876a-6bd268ae64e4","Type":"ContainerStarted","Data":"b35aef7be6c025ead2bae540f15414ec4f3835dc822ee0baf627c8e5bb1505b5"}
Nov 01 22:24:54.909958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:24:54.909924    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6nbtd" event={"ID":"9ced8467-be14-4a8a-a3ae-15566f579cc2","Type":"ContainerStarted","Data":"9a2b617f2009bf54595fd218e79dbcb8655173923ba03d407f76ed979bb68461"}
Nov 01 22:25:03.943016 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:25:03.942982    4837 generic.go:356] "Generic (PLEG): container finished" podID="5b6492e2-6280-45e6-80df-537fa2489fdf" containerID="d58508ac467cb86db290f537e090958567b258102af460e71a36060d98b398a9" exitCode=0
Nov 01 22:25:03.943430 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:25:03.943062    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb" event={"ID":"5b6492e2-6280-45e6-80df-537fa2489fdf","Type":"ContainerDied","Data":"d58508ac467cb86db290f537e090958567b258102af460e71a36060d98b398a9"}
Nov 01 22:25:03.943430 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:25:03.943347    4837 scope.go:122] "RemoveContainer" containerID="d58508ac467cb86db290f537e090958567b258102af460e71a36060d98b398a9"
Nov 01 22:25:04.948794 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:25:04.948757    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hdxcb" event={"ID":"5b6492e2-6280-45e6-80df-537fa2489fdf","Type":"ContainerStarted","Data":"28d1015feb15fafcec193f647a443b41bdeebf344899ca131b1375041f6c2142"}
Nov 01 22:25:18.810018 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:25:18.809983    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:25:48.810885 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:25:48.810812    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:26:18.810743 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:26:18.810703    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:26:48.810561 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:26:48.810522    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:27:18.810812 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:27:18.810731    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:27:43.284932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.284896    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"]
Nov 01 22:27:43.285439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.285246    4837 cpu_manager.go:414] "containerMap: removing container" podUID="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" containerName="registry"
Nov 01 22:27:43.285439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.285260    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" containerName="registry"
Nov 01 22:27:43.285439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.285278    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f69ab461-32d6-44ee-912e-95c4c27d6696" containerName="registry"
Nov 01 22:27:43.285439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.285287    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69ab461-32d6-44ee-912e-95c4c27d6696" containerName="registry"
Nov 01 22:27:43.285439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.285362    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69ab461-32d6-44ee-912e-95c4c27d6696" containerName="registry"
Nov 01 22:27:43.285439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.285379    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9a27c9-af1d-47a8-96d1-e2618c5c0900" containerName="registry"
Nov 01 22:27:43.288286 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.288266    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.340073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.340033    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 22:27:43.343157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.343140    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 22:27:43.343253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.343172    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 22:27:43.364898 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.364861    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.364992 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.364913    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.365054 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.365012    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d088a96b-ff8f-4088-99a7-9d10fac193bc-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.365097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.365036    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.365097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.365074    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxjdl\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-kube-api-access-gxjdl\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.365157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.365130    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.365190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.365163    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.384409 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.384386    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 22:27:43.384500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.384415    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b55rk\""
Nov 01 22:27:43.385490 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.385474    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"]
Nov 01 22:27:43.466088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.466060    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.466177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.466094    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.466177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.466113    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d088a96b-ff8f-4088-99a7-9d10fac193bc-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.466258 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.466229    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.466300 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.466267    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjdl\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-kube-api-access-gxjdl\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.466351 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.466312    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.466399 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.466357    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.466730 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.466705    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.468216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.468198    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d088a96b-ff8f-4088-99a7-9d10fac193bc-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.468514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.468498    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.468839 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.468817    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.468944 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.468925    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.520633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.520610    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjdl\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-kube-api-access-gxjdl\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.521150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.521129    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gf59t\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:43.599253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:43.599206    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:44.012121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:44.012098    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"]
Nov 01 22:27:44.014683 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:27:44.014652    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd088a96b_ff8f_4088_99a7_9d10fac193bc.slice/crio-a11c986a896420324c66ca78098a9cf5abdfb157a4c9e47de45c760b407379bc WatchSource:0}: Error finding container a11c986a896420324c66ca78098a9cf5abdfb157a4c9e47de45c760b407379bc: Status 404 returned error can't find the container with id a11c986a896420324c66ca78098a9cf5abdfb157a4c9e47de45c760b407379bc
Nov 01 22:27:44.509720 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:44.509684    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t" event={"ID":"d088a96b-ff8f-4088-99a7-9d10fac193bc","Type":"ContainerStarted","Data":"a11c986a896420324c66ca78098a9cf5abdfb157a4c9e47de45c760b407379bc"}
Nov 01 22:27:48.445828 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:48.445793    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:27:48.446178 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:48.445858    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:27:48.528188 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:48.528160    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t" event={"ID":"d088a96b-ff8f-4088-99a7-9d10fac193bc","Type":"ContainerStarted","Data":"e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537"}
Nov 01 22:27:48.528331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:48.528273    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:27:48.669214 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:48.669152    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t" podStartSLOduration=1.240546697 podStartE2EDuration="5.669136097s" totalImagesPullingTime="4.4285894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:48.668371384 +0000 UTC m=+250.257570445" watchObservedRunningTime="2026-11-01 22:27:48.669136097 +0000 UTC m=+250.258335158"
Nov 01 22:27:48.811135 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:27:48.811066    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:27:49.533359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:49.533326    4837 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-gf59t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:27:49.533720 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:49.533385    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t" podUID="d088a96b-ff8f-4088-99a7-9d10fac193bc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:27:52.532496 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:27:52.532469    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:28:18.810407 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:28:18.810366    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:28:38.538410 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:38.538378    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:28:38.539012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:38.538838    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:28:41.372964 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.372929    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"]
Nov 01 22:28:41.382855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.382835    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.474746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.474719    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7wbc\" (UniqueName: \"kubernetes.io/projected/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-kube-api-access-b7wbc\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.474875 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.474770    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.474875 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.474790    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.474995 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.474876    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.474995 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.474892    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.474995 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.474938    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.483315 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.483292    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"]
Nov 01 22:28:41.508216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.508193    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 22:28:41.508318 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.508245    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 22:28:41.508386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.508326    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 22:28:41.508386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.508367    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 22:28:41.508386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.508379    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 22:28:41.508490 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.508399    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jfzq8\""
Nov 01 22:28:41.576242 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.576220    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7wbc\" (UniqueName: \"kubernetes.io/projected/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-kube-api-access-b7wbc\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.576374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.576275    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.576374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.576299    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.576374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.576334    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.576374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.576354    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.576374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.576369    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.576785 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.576755    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.578919 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.578895    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.578999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.578925    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.579062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.579027    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.579278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.579251    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.631280 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.631221    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7wbc\" (UniqueName: \"kubernetes.io/projected/b0e1143f-9b2d-4903-a25c-0b86c7cecde9-kube-api-access-b7wbc\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hmhc4\" (UID: \"b0e1143f-9b2d-4903-a25c-0b86c7cecde9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:41.697147 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:41.697119    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:42.234731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:42.234704    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"]
Nov 01 22:28:42.236804 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:28:42.236770    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e1143f_9b2d_4903_a25c_0b86c7cecde9.slice/crio-082078270bc37bfa5c5efa310813e9ca1798aa39467296925a94dfce54b2c362 WatchSource:0}: Error finding container 082078270bc37bfa5c5efa310813e9ca1798aa39467296925a94dfce54b2c362: Status 404 returned error can't find the container with id 082078270bc37bfa5c5efa310813e9ca1798aa39467296925a94dfce54b2c362
Nov 01 22:28:42.716673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:42.716634    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4" event={"ID":"b0e1143f-9b2d-4903-a25c-0b86c7cecde9","Type":"ContainerStarted","Data":"082078270bc37bfa5c5efa310813e9ca1798aa39467296925a94dfce54b2c362"}
Nov 01 22:28:46.734424 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:46.734394    4837 generic.go:356] "Generic (PLEG): container finished" podID="b0e1143f-9b2d-4903-a25c-0b86c7cecde9" containerID="89b1c90f443465268c933c6f86c65b68f49e5cf4f3094d1c046f3a1019458b0d" exitCode=0
Nov 01 22:28:46.734819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:46.734476    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4" event={"ID":"b0e1143f-9b2d-4903-a25c-0b86c7cecde9","Type":"ContainerDied","Data":"89b1c90f443465268c933c6f86c65b68f49e5cf4f3094d1c046f3a1019458b0d"}
Nov 01 22:28:47.739938 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:47.739906    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4" event={"ID":"b0e1143f-9b2d-4903-a25c-0b86c7cecde9","Type":"ContainerStarted","Data":"bf49776c7e2ca1e3fa1eec0aac5ef724515b9585d17c13708e4938e3cd92011e"}
Nov 01 22:28:47.740316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:47.740035    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:28:47.957542 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:47.957491    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4" podStartSLOduration=1.738657371 podStartE2EDuration="6.957476086s" totalImagesPullingTime="4.756598994s" totalInitContainerRuntime="462.219721ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:47.956495205 +0000 UTC m=+309.545694267" watchObservedRunningTime="2026-11-01 22:28:47.957476086 +0000 UTC m=+309.546675147"
Nov 01 22:28:48.811249 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:28:48.811214    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:28:58.746467 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:28:58.746437    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4"
Nov 01 22:29:18.605807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.605767    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"]
Nov 01 22:29:18.608414 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.608391    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.651535 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.651510    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4vw\" (UniqueName: \"kubernetes.io/projected/cb8723c9-c41d-483b-9805-df01a4a041d3-kube-api-access-hv4vw\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.651661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.651542    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8723c9-c41d-483b-9805-df01a4a041d3-tls-certs\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.651661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.651566    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb8723c9-c41d-483b-9805-df01a4a041d3-tmp\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.679623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.679592    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 22:29:18.703143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.703118    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7w5mv\""
Nov 01 22:29:18.703143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.703140    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 22:29:18.727248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.727222    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"]
Nov 01 22:29:18.753002 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.752973    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4vw\" (UniqueName: \"kubernetes.io/projected/cb8723c9-c41d-483b-9805-df01a4a041d3-kube-api-access-hv4vw\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.753119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.753005    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8723c9-c41d-483b-9805-df01a4a041d3-tls-certs\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.753119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.753063    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb8723c9-c41d-483b-9805-df01a4a041d3-tmp\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.755400 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.755380    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb8723c9-c41d-483b-9805-df01a4a041d3-tmp\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.755546 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.755530    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8723c9-c41d-483b-9805-df01a4a041d3-tls-certs\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.806280 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.806257    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4vw\" (UniqueName: \"kubernetes.io/projected/cb8723c9-c41d-483b-9805-df01a4a041d3-kube-api-access-hv4vw\") pod \"kube-auth-proxy-59b78b6bf6-drlpm\" (UID: \"cb8723c9-c41d-483b-9805-df01a4a041d3\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:18.813365 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:29:18.813333    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:29:18.919329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:18.919309    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"
Nov 01 22:29:19.280383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:19.280357    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm"]
Nov 01 22:29:19.283086 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:29:19.283054    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8723c9_c41d_483b_9805_df01a4a041d3.slice/crio-53adc0fcfa9c357e7f5a6f2d4473c1164e86c3b105a8244ea5f3392d7e7aa09f WatchSource:0}: Error finding container 53adc0fcfa9c357e7f5a6f2d4473c1164e86c3b105a8244ea5f3392d7e7aa09f: Status 404 returned error can't find the container with id 53adc0fcfa9c357e7f5a6f2d4473c1164e86c3b105a8244ea5f3392d7e7aa09f
Nov 01 22:29:19.284828 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:19.284812    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:29:19.859534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:19.859491    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm" event={"ID":"cb8723c9-c41d-483b-9805-df01a4a041d3","Type":"ContainerStarted","Data":"53adc0fcfa9c357e7f5a6f2d4473c1164e86c3b105a8244ea5f3392d7e7aa09f"}
Nov 01 22:29:22.875052 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:22.875017    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm" event={"ID":"cb8723c9-c41d-483b-9805-df01a4a041d3","Type":"ContainerStarted","Data":"700f913980c704dccf224e41c8c26cae02375c80413b87bd66cd6742f16ff530"}
Nov 01 22:29:22.982770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:22.982716    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-drlpm" podStartSLOduration=1.895914416 podStartE2EDuration="4.982699855s" totalImagesPullingTime="3.086785439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:22.981984917 +0000 UTC m=+344.571183978" watchObservedRunningTime="2026-11-01 22:29:22.982699855 +0000 UTC m=+344.571898920"
Nov 01 22:29:27.499203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.499166    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:29:27.503473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.503453    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:27.558098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.558072    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:29:27.603305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.603275    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 22:29:27.603305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.603283    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 22:29:27.603461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.603303    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 22:29:27.603461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.603335    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qf6wd\""
Nov 01 22:29:27.630021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.629992    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c058156-64d8-4831-9056-3d62bc7f0bff-config-file\") pod \"limitador-limitador-7c9b87d574-lhprg\" (UID: \"4c058156-64d8-4831-9056-3d62bc7f0bff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:27.630161 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.630029    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztq64\" (UniqueName: \"kubernetes.io/projected/4c058156-64d8-4831-9056-3d62bc7f0bff-kube-api-access-ztq64\") pod \"limitador-limitador-7c9b87d574-lhprg\" (UID: \"4c058156-64d8-4831-9056-3d62bc7f0bff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:27.730455 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.730423    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c058156-64d8-4831-9056-3d62bc7f0bff-config-file\") pod \"limitador-limitador-7c9b87d574-lhprg\" (UID: \"4c058156-64d8-4831-9056-3d62bc7f0bff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:27.730455 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.730457    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztq64\" (UniqueName: \"kubernetes.io/projected/4c058156-64d8-4831-9056-3d62bc7f0bff-kube-api-access-ztq64\") pod \"limitador-limitador-7c9b87d574-lhprg\" (UID: \"4c058156-64d8-4831-9056-3d62bc7f0bff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:27.731059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.731022    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c058156-64d8-4831-9056-3d62bc7f0bff-config-file\") pod \"limitador-limitador-7c9b87d574-lhprg\" (UID: \"4c058156-64d8-4831-9056-3d62bc7f0bff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:27.793552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.793490    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztq64\" (UniqueName: \"kubernetes.io/projected/4c058156-64d8-4831-9056-3d62bc7f0bff-kube-api-access-ztq64\") pod \"limitador-limitador-7c9b87d574-lhprg\" (UID: \"4c058156-64d8-4831-9056-3d62bc7f0bff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:27.798518 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.798486    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:29:27.815404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:27.815379    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:28.177682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:28.177657    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:29:28.180263 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:29:28.180234    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c058156_64d8_4831_9056_3d62bc7f0bff.slice/crio-efc621284cd1bb934630180f822e2d47d87f7215cf97ae5d0b62e3710a3267f2 WatchSource:0}: Error finding container efc621284cd1bb934630180f822e2d47d87f7215cf97ae5d0b62e3710a3267f2: Status 404 returned error can't find the container with id efc621284cd1bb934630180f822e2d47d87f7215cf97ae5d0b62e3710a3267f2
Nov 01 22:29:28.901684 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:28.901649    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg" event={"ID":"4c058156-64d8-4831-9056-3d62bc7f0bff","Type":"ContainerStarted","Data":"efc621284cd1bb934630180f822e2d47d87f7215cf97ae5d0b62e3710a3267f2"}
Nov 01 22:29:32.920664 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:32.920633    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg" event={"ID":"4c058156-64d8-4831-9056-3d62bc7f0bff","Type":"ContainerStarted","Data":"9cf9be08726572118669b9db2f21b2e663c9bca486d1e56e7a75a9196bd9db50"}
Nov 01 22:29:32.921090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:32.920782    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:33.039577 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:33.039534    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg" podStartSLOduration=2.19735143 podStartE2EDuration="6.039522827s" totalImagesPullingTime="3.842171397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:33.038910944 +0000 UTC m=+354.628110005" watchObservedRunningTime="2026-11-01 22:29:33.039522827 +0000 UTC m=+354.628721884"
Nov 01 22:29:43.925747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:29:43.925723    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhprg"
Nov 01 22:29:48.810314 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:29:48.810275    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:30:18.810565 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:30:18.810488    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:30:48.811347 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:30:48.811301    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:30:49.910725 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:49.910698    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"]
Nov 01 22:30:49.913994 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:49.913969    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.010866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.010842    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"]
Nov 01 22:30:50.022471 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.022449    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.022673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.022475    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadcdb38-ad92-4d97-a143-7753934cd55b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.022673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.022504    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb7fr\" (UniqueName: \"kubernetes.io/projected/aadcdb38-ad92-4d97-a143-7753934cd55b-kube-api-access-hb7fr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.022673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.022606    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.022673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.022624    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.022673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.022644    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.022673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.022658    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.123784 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.123755    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.123901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.123788    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.123901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.123803    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.123901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.123833    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.123901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.123850    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadcdb38-ad92-4d97-a143-7753934cd55b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.124133 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.123994    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb7fr\" (UniqueName: \"kubernetes.io/projected/aadcdb38-ad92-4d97-a143-7753934cd55b-kube-api-access-hb7fr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.124133 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.124105    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.124642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.124619    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.126095 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.126072    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadcdb38-ad92-4d97-a143-7753934cd55b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.126389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.126361    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.126610 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.126595    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.126651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.126634    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.177284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.177255    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb7fr\" (UniqueName: \"kubernetes.io/projected/aadcdb38-ad92-4d97-a143-7753934cd55b-kube-api-access-hb7fr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.177372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.177325    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadcdb38-ad92-4d97-a143-7753934cd55b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5dx5\" (UID: \"aadcdb38-ad92-4d97-a143-7753934cd55b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.226270 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.226243    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:50.637184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.637158    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"]
Nov 01 22:30:50.639163 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:30:50.639132    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadcdb38_ad92_4d97_a143_7753934cd55b.slice/crio-ade665fd18f14b3b3f54c5486b2854e4ad5a67b6f7a4b7cbe44d61334f262c77 WatchSource:0}: Error finding container ade665fd18f14b3b3f54c5486b2854e4ad5a67b6f7a4b7cbe44d61334f262c77: Status 404 returned error can't find the container with id ade665fd18f14b3b3f54c5486b2854e4ad5a67b6f7a4b7cbe44d61334f262c77
Nov 01 22:30:50.641296 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.641265    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:30:50.641383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:50.641323    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:30:51.233326 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:51.233293    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5" event={"ID":"aadcdb38-ad92-4d97-a143-7753934cd55b","Type":"ContainerStarted","Data":"221e4a139dd42f6ad0cb3f1a8573d3ae73532f7ee004575152b1bdb5ba4a6cb8"}
Nov 01 22:30:51.233705 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:51.233342    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5" event={"ID":"aadcdb38-ad92-4d97-a143-7753934cd55b","Type":"ContainerStarted","Data":"ade665fd18f14b3b3f54c5486b2854e4ad5a67b6f7a4b7cbe44d61334f262c77"}
Nov 01 22:30:51.233705 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:51.233430    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:51.234868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:51.234843    4837 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-l5dx5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:30:51.234974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:51.234892    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5" podUID="aadcdb38-ad92-4d97-a143-7753934cd55b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:30:51.342353 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:51.342310    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5" podStartSLOduration=2.34229791 podStartE2EDuration="2.34229791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:51.340992399 +0000 UTC m=+432.930191461" watchObservedRunningTime="2026-11-01 22:30:51.34229791 +0000 UTC m=+432.931496971"
Nov 01 22:30:52.237676 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.237646    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5dx5"
Nov 01 22:30:52.441552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.441520    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"]
Nov 01 22:30:52.442013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.441950    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t" podUID="d088a96b-ff8f-4088-99a7-9d10fac193bc" containerName="discovery" containerID="cri-o://e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537" gracePeriod=30
Nov 01 22:30:52.532336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.532311    4837 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-gf59t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:8080/ready\": dial tcp 10.134.0.24:8080: connect: connection refused" start-of-body=
Nov 01 22:30:52.532440 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.532368    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t" podUID="d088a96b-ff8f-4088-99a7-9d10fac193bc" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.24:8080/ready\": dial tcp 10.134.0.24:8080: connect: connection refused"
Nov 01 22:30:52.688467 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.688436    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:30:52.743586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.743557    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-dns-cert\") pod \"d088a96b-ff8f-4088-99a7-9d10fac193bc\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") "
Nov 01 22:30:52.743685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.743590    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-cacerts\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-cacerts\") pod \"d088a96b-ff8f-4088-99a7-9d10fac193bc\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") "
Nov 01 22:30:52.743685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.743637    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d088a96b-ff8f-4088-99a7-9d10fac193bc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d088a96b-ff8f-4088-99a7-9d10fac193bc-local-certs\") pod \"d088a96b-ff8f-4088-99a7-9d10fac193bc\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") "
Nov 01 22:30:52.743803 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.743760    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-kube-api-access-gxjdl\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-kube-api-access-gxjdl\") pod \"d088a96b-ff8f-4088-99a7-9d10fac193bc\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") "
Nov 01 22:30:52.743858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.743804    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-token\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-token\") pod \"d088a96b-ff8f-4088-99a7-9d10fac193bc\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") "
Nov 01 22:30:52.743858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.743845    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-ca-configmap\") pod \"d088a96b-ff8f-4088-99a7-9d10fac193bc\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") "
Nov 01 22:30:52.743953 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.743910    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-kubeconfig\") pod \"d088a96b-ff8f-4088-99a7-9d10fac193bc\" (UID: \"d088a96b-ff8f-4088-99a7-9d10fac193bc\") "
Nov 01 22:30:52.744361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.744331    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-ca-configmap" pod "d088a96b-ff8f-4088-99a7-9d10fac193bc" (UID: "d088a96b-ff8f-4088-99a7-9d10fac193bc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:52.746373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.746347    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-kubeconfig" pod "d088a96b-ff8f-4088-99a7-9d10fac193bc" (UID: "d088a96b-ff8f-4088-99a7-9d10fac193bc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:52.746475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.746390    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-dns-cert" pod "d088a96b-ff8f-4088-99a7-9d10fac193bc" (UID: "d088a96b-ff8f-4088-99a7-9d10fac193bc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:52.746475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.746391    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d088a96b-ff8f-4088-99a7-9d10fac193bc-local-certs" pod "d088a96b-ff8f-4088-99a7-9d10fac193bc" (UID: "d088a96b-ff8f-4088-99a7-9d10fac193bc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:52.746475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.746453    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-cacerts" pod "d088a96b-ff8f-4088-99a7-9d10fac193bc" (UID: "d088a96b-ff8f-4088-99a7-9d10fac193bc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:52.746475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.746460    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-token" pod "d088a96b-ff8f-4088-99a7-9d10fac193bc" (UID: "d088a96b-ff8f-4088-99a7-9d10fac193bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:52.746475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.746466    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-kube-api-access-gxjdl" pod "d088a96b-ff8f-4088-99a7-9d10fac193bc" (UID: "d088a96b-ff8f-4088-99a7-9d10fac193bc"). InnerVolumeSpecName "kube-api-access-gxjdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:52.844566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.844505    4837 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d088a96b-ff8f-4088-99a7-9d10fac193bc-local-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:52.844566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.844531    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxjdl\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-kube-api-access-gxjdl\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:52.844566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.844541    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:52.844566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.844550    4837 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-ca-configmap\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:52.844566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.844559    4837 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-kubeconfig\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:52.844566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.844567    4837 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-istio-csr-dns-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:52.844835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:52.844576    4837 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d088a96b-ff8f-4088-99a7-9d10fac193bc-cacerts\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:53.243071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.243021    4837 generic.go:356] "Generic (PLEG): container finished" podID="d088a96b-ff8f-4088-99a7-9d10fac193bc" containerID="e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537" exitCode=0
Nov 01 22:30:53.243533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.243089    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t" event={"ID":"d088a96b-ff8f-4088-99a7-9d10fac193bc","Type":"ContainerDied","Data":"e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537"}
Nov 01 22:30:53.243533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.243115    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"
Nov 01 22:30:53.243533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.243143    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t" event={"ID":"d088a96b-ff8f-4088-99a7-9d10fac193bc","Type":"ContainerDied","Data":"a11c986a896420324c66ca78098a9cf5abdfb157a4c9e47de45c760b407379bc"}
Nov 01 22:30:53.243533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.243168    4837 scope.go:122] "RemoveContainer" containerID="e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537"
Nov 01 22:30:53.243876 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.243801    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wt2xw"]
Nov 01 22:30:53.244204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.244174    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d088a96b-ff8f-4088-99a7-9d10fac193bc" containerName="discovery"
Nov 01 22:30:53.244204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.244196    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d088a96b-ff8f-4088-99a7-9d10fac193bc" containerName="discovery"
Nov 01 22:30:53.244427 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.244333    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d088a96b-ff8f-4088-99a7-9d10fac193bc" containerName="discovery"
Nov 01 22:30:53.247473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.247454    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:53.252023 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.252003    4837 scope.go:122] "RemoveContainer" containerID="e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537"
Nov 01 22:30:53.252322 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:30:53.252299    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537\": container with ID starting with e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537 not found: ID does not exist" containerID="e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537"
Nov 01 22:30:53.252415 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.252332    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537"} err="failed to get container status \"e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537\": rpc error: code = NotFound desc = could not find container \"e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537\": container with ID starting with e3f055fd27bba8592c9102081dcf8a4966b5bb995e29baab65a45a04b2c40537 not found: ID does not exist"
Nov 01 22:30:53.296152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.296128    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 22:30:53.296276 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.296265    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 22:30:53.301482 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.301462    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wt2xw"]
Nov 01 22:30:53.304422 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.304402    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 01 22:30:53.304561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.304546    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wqszf\""
Nov 01 22:30:53.348873 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.348848    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c69dm\" (UniqueName: \"kubernetes.io/projected/4940c23d-3994-46bb-bea1-7303c8a50498-kube-api-access-c69dm\") pod \"kserve-controller-manager-776cc66d47-wt2xw\" (UID: \"4940c23d-3994-46bb-bea1-7303c8a50498\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:53.349096 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.349078    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4940c23d-3994-46bb-bea1-7303c8a50498-cert\") pod \"kserve-controller-manager-776cc66d47-wt2xw\" (UID: \"4940c23d-3994-46bb-bea1-7303c8a50498\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:53.449396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.449370    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4940c23d-3994-46bb-bea1-7303c8a50498-cert\") pod \"kserve-controller-manager-776cc66d47-wt2xw\" (UID: \"4940c23d-3994-46bb-bea1-7303c8a50498\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:53.449500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.449414    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c69dm\" (UniqueName: \"kubernetes.io/projected/4940c23d-3994-46bb-bea1-7303c8a50498-kube-api-access-c69dm\") pod \"kserve-controller-manager-776cc66d47-wt2xw\" (UID: \"4940c23d-3994-46bb-bea1-7303c8a50498\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:53.451793 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.451770    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4940c23d-3994-46bb-bea1-7303c8a50498-cert\") pod \"kserve-controller-manager-776cc66d47-wt2xw\" (UID: \"4940c23d-3994-46bb-bea1-7303c8a50498\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:53.502123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.502077    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"]
Nov 01 22:30:53.508532 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.508511    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c69dm\" (UniqueName: \"kubernetes.io/projected/4940c23d-3994-46bb-bea1-7303c8a50498-kube-api-access-c69dm\") pod \"kserve-controller-manager-776cc66d47-wt2xw\" (UID: \"4940c23d-3994-46bb-bea1-7303c8a50498\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:53.509891 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.509871    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gf59t"]
Nov 01 22:30:53.550741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.550717    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-26jmv"]
Nov 01 22:30:53.554008 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.553992    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.562292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.562272    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:53.602391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.602369    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-w7khz\""
Nov 01 22:30:53.602512 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.602429    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 01 22:30:53.602564 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.602547    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 01 22:30:53.611138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.609974    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-26jmv"]
Nov 01 22:30:53.651035 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.651010    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5861219-3f6b-475c-810b-168da625ec9a-cert\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.651146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.651057    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5861219-3f6b-475c-810b-168da625ec9a-metrics-certs\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.651183 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.651165    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j9qw\" (UniqueName: \"kubernetes.io/projected/b5861219-3f6b-475c-810b-168da625ec9a-kube-api-access-9j9qw\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.752535 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.752484    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5861219-3f6b-475c-810b-168da625ec9a-cert\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.752535 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.752509    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5861219-3f6b-475c-810b-168da625ec9a-metrics-certs\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.752661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.752542    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j9qw\" (UniqueName: \"kubernetes.io/projected/b5861219-3f6b-475c-810b-168da625ec9a-kube-api-access-9j9qw\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.754737 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.754716    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5861219-3f6b-475c-810b-168da625ec9a-cert\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.754828 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.754784    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5861219-3f6b-475c-810b-168da625ec9a-metrics-certs\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.805729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.805709    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j9qw\" (UniqueName: \"kubernetes.io/projected/b5861219-3f6b-475c-810b-168da625ec9a-kube-api-access-9j9qw\") pod \"odh-model-controller-769875db56-26jmv\" (UID: \"b5861219-3f6b-475c-810b-168da625ec9a\") " pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.866531 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.866503    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:53.930960 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:30:53.928443    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4940c23d_3994_46bb_bea1_7303c8a50498.slice/crio-8f3408afaed88f3bc9aa944ef909a8a498273822c00e81fcb66f507ea1f0873a WatchSource:0}: Error finding container 8f3408afaed88f3bc9aa944ef909a8a498273822c00e81fcb66f507ea1f0873a: Status 404 returned error can't find the container with id 8f3408afaed88f3bc9aa944ef909a8a498273822c00e81fcb66f507ea1f0873a
Nov 01 22:30:53.930960 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:53.930643    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wt2xw"]
Nov 01 22:30:54.227484 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:54.227443    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-26jmv"]
Nov 01 22:30:54.230094 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:30:54.230067    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5861219_3f6b_475c_810b_168da625ec9a.slice/crio-b63a654bf633e490a318fcb4cb109beb56fd10fda2666c6cf270db4ba5a8dfb7 WatchSource:0}: Error finding container b63a654bf633e490a318fcb4cb109beb56fd10fda2666c6cf270db4ba5a8dfb7: Status 404 returned error can't find the container with id b63a654bf633e490a318fcb4cb109beb56fd10fda2666c6cf270db4ba5a8dfb7
Nov 01 22:30:54.248471 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:54.248442    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw" event={"ID":"4940c23d-3994-46bb-bea1-7303c8a50498","Type":"ContainerStarted","Data":"8f3408afaed88f3bc9aa944ef909a8a498273822c00e81fcb66f507ea1f0873a"}
Nov 01 22:30:54.250420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:54.250400    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-26jmv" event={"ID":"b5861219-3f6b-475c-810b-168da625ec9a","Type":"ContainerStarted","Data":"b63a654bf633e490a318fcb4cb109beb56fd10fda2666c6cf270db4ba5a8dfb7"}
Nov 01 22:30:54.522090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:54.522008    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d088a96b-ff8f-4088-99a7-9d10fac193bc" path="/var/lib/kubelet/pods/d088a96b-ff8f-4088-99a7-9d10fac193bc/volumes"
Nov 01 22:30:54.859310 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:54.859244    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:30:57.266499 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:57.266457    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw" event={"ID":"4940c23d-3994-46bb-bea1-7303c8a50498","Type":"ContainerStarted","Data":"5e67a013a8423f79334a7168e316467a9bf504468ea3d441eb5c3b34cd8a1c59"}
Nov 01 22:30:57.266943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:57.266666    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:30:57.373118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:57.373064    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw" podStartSLOduration=1.527692473 podStartE2EDuration="4.373032191s" totalImagesPullingTime="2.845339718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:57.372402728 +0000 UTC m=+438.961601808" watchObservedRunningTime="2026-11-01 22:30:57.373032191 +0000 UTC m=+438.962231249"
Nov 01 22:30:58.271723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:58.271686    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-26jmv" event={"ID":"b5861219-3f6b-475c-810b-168da625ec9a","Type":"ContainerStarted","Data":"24078244e74210f922fe20ff16ad90f2e8848b6a9bf38790d9283bf3c779f0e0"}
Nov 01 22:30:58.272114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:58.271767    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:30:58.380221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:30:58.380180    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-26jmv" podStartSLOduration=1.727447934 podStartE2EDuration="5.380165343s" totalImagesPullingTime="3.652717409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:58.379568623 +0000 UTC m=+439.968767681" watchObservedRunningTime="2026-11-01 22:30:58.380165343 +0000 UTC m=+439.969364405"
Nov 01 22:31:09.278177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:31:09.278148    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-26jmv"
Nov 01 22:31:18.811424 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:31:18.811386    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:31:28.277859 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:31:28.277830    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wt2xw"
Nov 01 22:31:48.811121 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:31:48.811027    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:32:18.810956 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:32:18.810922    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:32:48.810961 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:32:48.810925    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:32:58.828018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.827982    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"]
Nov 01 22:32:58.830675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.830653    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.865616 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.865583    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.865743 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.865624    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.865824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.865744    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.865824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.865769    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwrx\" (UniqueName: \"kubernetes.io/projected/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-kube-api-access-dhwrx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.865978 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.865827    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.865978 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.865860    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.893640 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.893616    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 22:32:58.925866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.925844    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 22:32:58.925866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.925847    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 22:32:58.926100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.925972    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4662b\""
Nov 01 22:32:58.927303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.927280    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"]
Nov 01 22:32:58.966227 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966200    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.966322 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966234    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.966322 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966262    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.966322 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966278    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwrx\" (UniqueName: \"kubernetes.io/projected/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-kube-api-access-dhwrx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.966322 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966303    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.966467 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966326    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.966644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966624    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.966704 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966652    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.966761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.966700    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.968478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.968458    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:58.968709 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:58.968694    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:59.019509 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:59.019486    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwrx\" (UniqueName: \"kubernetes.io/projected/87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9-kube-api-access-dhwrx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq565\" (UID: \"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:59.142096 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:59.142017    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:32:59.515518 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:59.515488    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"]
Nov 01 22:32:59.517766 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:32:59.517738    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b00ef5_41fb_4dbd_834e_c2cb76f1e6f9.slice/crio-b257056811648c47b9bd1a01c2efbb5a0b7d3ce3a11bbb6385ad6f008e065275 WatchSource:0}: Error finding container b257056811648c47b9bd1a01c2efbb5a0b7d3ce3a11bbb6385ad6f008e065275: Status 404 returned error can't find the container with id b257056811648c47b9bd1a01c2efbb5a0b7d3ce3a11bbb6385ad6f008e065275
Nov 01 22:32:59.751319 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:32:59.751290    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565" event={"ID":"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9","Type":"ContainerStarted","Data":"b257056811648c47b9bd1a01c2efbb5a0b7d3ce3a11bbb6385ad6f008e065275"}
Nov 01 22:33:01.762181 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:01.762100    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565" event={"ID":"87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9","Type":"ContainerStarted","Data":"9743da9bd8b1e86167b4aa1128bf6fc093ff1f0939fbe890dc4325e34e3e64e0"}
Nov 01 22:33:01.870983 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:01.870939    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565" podStartSLOduration=1.917387068 podStartE2EDuration="3.870924599s" totalImagesPullingTime="1.953537531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:01.870074596 +0000 UTC m=+563.459273653" watchObservedRunningTime="2026-11-01 22:33:01.870924599 +0000 UTC m=+563.460123657"
Nov 01 22:33:02.142643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:02.142558    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:33:02.211899 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:02.211861    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:33:06.198690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.198657    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"]
Nov 01 22:33:06.202289 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.202272    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.214604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.214583    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.214680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.214615    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfc8\" (UniqueName: \"kubernetes.io/projected/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-kube-api-access-bkfc8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.214723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.214703    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.214763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.214739    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.214763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.214754    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.214887 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.214868    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.299975 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.299947    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 22:33:06.301204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.301187    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"]
Nov 01 22:33:06.315481 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.315451    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.315583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.315484    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.315583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.315503    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.315583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.315543    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.315694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.315595    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.315694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.315629    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfc8\" (UniqueName: \"kubernetes.io/projected/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-kube-api-access-bkfc8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.315963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.315943    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.316099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.316001    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.316099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.316006    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.317734 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.317713    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.318021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.318004    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.368473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.368449    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfc8\" (UniqueName: \"kubernetes.io/projected/7ea44462-9d6a-42ec-ae71-4f8b3567f7b1-kube-api-access-bkfc8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4jbsp\" (UID: \"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.514290 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.514210    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:06.877348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:06.877310    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"]
Nov 01 22:33:06.879457 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:33:06.879412    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea44462_9d6a_42ec_ae71_4f8b3567f7b1.slice/crio-fd911f926afc152672029a43eaa119998e543705858beaae3af4bb49ab2767a9 WatchSource:0}: Error finding container fd911f926afc152672029a43eaa119998e543705858beaae3af4bb49ab2767a9: Status 404 returned error can't find the container with id fd911f926afc152672029a43eaa119998e543705858beaae3af4bb49ab2767a9
Nov 01 22:33:07.789865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:07.789786    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp" event={"ID":"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1","Type":"ContainerStarted","Data":"a493bb2b4ecdc1063a7b6cf0f01e35ca012e74d6c4a6c7fa3e9698a16e40f634"}
Nov 01 22:33:07.789865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:07.789834    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp" event={"ID":"7ea44462-9d6a-42ec-ae71-4f8b3567f7b1","Type":"ContainerStarted","Data":"fd911f926afc152672029a43eaa119998e543705858beaae3af4bb49ab2767a9"}
Nov 01 22:33:07.897497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:07.897448    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp" podStartSLOduration=1.29909956 podStartE2EDuration="1.897434243s" totalImagesPullingTime="598.334683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:07.896936698 +0000 UTC m=+569.486135760" watchObservedRunningTime="2026-11-01 22:33:07.897434243 +0000 UTC m=+569.486633303"
Nov 01 22:33:08.514494 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:08.514466    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:09.142522 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:09.142486    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:33:09.154735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:09.154712    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:33:09.809645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:09.809616    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565"
Nov 01 22:33:12.587477 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:12.587447    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:33:13.856007 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.855978    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-t294z"]
Nov 01 22:33:13.860402 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.860372    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.871248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.871223    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nncd9\" (UniqueName: \"kubernetes.io/projected/ef383e25-1da9-4495-85d4-e22fbcd68022-kube-api-access-nncd9\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.871361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.871265    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.871361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.871290    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-home\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.871445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.871424    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.871499 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.871483    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.871536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.871522    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef383e25-1da9-4495-85d4-e22fbcd68022-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.909937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.909908    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 22:33:13.957762 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.957734    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-t294z"]
Nov 01 22:33:13.972670 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.972643    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-home\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.972810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.972708    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.972810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.972773    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.972810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.972796    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef383e25-1da9-4495-85d4-e22fbcd68022-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.972980 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.972828    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nncd9\" (UniqueName: \"kubernetes.io/projected/ef383e25-1da9-4495-85d4-e22fbcd68022-kube-api-access-nncd9\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.972980 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.972954    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.973180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.973157    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-home\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.973239 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.973190    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.973325 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.973307    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.974887 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.974868    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef383e25-1da9-4495-85d4-e22fbcd68022-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:13.975152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:13.975135    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef383e25-1da9-4495-85d4-e22fbcd68022-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:14.026435 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:14.026409    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nncd9\" (UniqueName: \"kubernetes.io/projected/ef383e25-1da9-4495-85d4-e22fbcd68022-kube-api-access-nncd9\") pod \"echo-simulator-kserve-64b6f8f96c-t294z\" (UID: \"ef383e25-1da9-4495-85d4-e22fbcd68022\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:14.174410 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:14.174378    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:14.538465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:14.538444    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-t294z"]
Nov 01 22:33:14.540463 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:33:14.540430    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef383e25_1da9_4495_85d4_e22fbcd68022.slice/crio-61b460470889b764cf11246bcb9c5ba90e39f7a4f6bc0028a5204f57ecc4705f WatchSource:0}: Error finding container 61b460470889b764cf11246bcb9c5ba90e39f7a4f6bc0028a5204f57ecc4705f: Status 404 returned error can't find the container with id 61b460470889b764cf11246bcb9c5ba90e39f7a4f6bc0028a5204f57ecc4705f
Nov 01 22:33:14.818966 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:14.818893    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z" event={"ID":"ef383e25-1da9-4495-85d4-e22fbcd68022","Type":"ContainerStarted","Data":"61b460470889b764cf11246bcb9c5ba90e39f7a4f6bc0028a5204f57ecc4705f"}
Nov 01 22:33:16.514781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:16.514746    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:16.527635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:16.527611    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:16.843914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:16.843848    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp"
Nov 01 22:33:17.838576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:17.838499    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z" event={"ID":"ef383e25-1da9-4495-85d4-e22fbcd68022","Type":"ContainerStarted","Data":"aa433a46eae5579b9e77ea8eb05b3b6ed596c12589ab788616f4fae2464285ce"}
Nov 01 22:33:17.946132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:17.946088    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z" podStartSLOduration=2.253698462 podStartE2EDuration="4.946075341s" totalImagesPullingTime="2.692376879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:17.944960789 +0000 UTC m=+579.534159851" watchObservedRunningTime="2026-11-01 22:33:17.946075341 +0000 UTC m=+579.535274423"
Nov 01 22:33:18.175454 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:18.175417    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:18.812101 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:33:18.811985    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:33:23.575441 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.575411    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"]
Nov 01 22:33:23.578800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.578780    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.627843 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.627823    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bv4kj\""
Nov 01 22:33:23.648143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648121    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.648245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648152    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a757e-f762-446b-aa18-5425bad1c560-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.648245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648170    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a757e-f762-446b-aa18-5425bad1c560-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.648245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648187    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.648348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648315    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl72l\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-kube-api-access-sl72l\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.648382 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648346    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.648414 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648378    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.648459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648446    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.648495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.648467    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.676251 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.676217    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"]
Nov 01 22:33:23.749717 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.749684    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.749869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.749729    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.749869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.749747    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.749869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.749781    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.749869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.749803    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a757e-f762-446b-aa18-5425bad1c560-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.749869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.749820    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a757e-f762-446b-aa18-5425bad1c560-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.749869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.749840    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.750263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.749880    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl72l\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-kube-api-access-sl72l\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.750263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.750084    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.750530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.750506    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.751446 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.751418    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.752058 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.752021    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a757e-f762-446b-aa18-5425bad1c560-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.752665 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.752634    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.758024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.753804    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a757e-f762-446b-aa18-5425bad1c560-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.758305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.758283    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.758832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.758815    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.764315 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.764291    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:33:23.805205 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.805184    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl72l\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-kube-api-access-sl72l\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.805304 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.805239    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-6z8hk\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:23.891626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:23.891557    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:24.139460 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.139430    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"]
Nov 01 22:33:24.143037 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.142988    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.174938 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.174885    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:24.189371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.189347    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:24.192108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.192090    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 22:33:24.198348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.198324    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"]
Nov 01 22:33:24.255968 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.255944    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.256090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.255973    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9gp\" (UniqueName: \"kubernetes.io/projected/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-kube-api-access-hs9gp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.256090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.255994    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.256168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.256089    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.256205 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.256185    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.256267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.256253    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.352390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.352371    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"]
Nov 01 22:33:24.354381 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:33:24.354356    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832a757e_f762_446b_aa18_5425bad1c560.slice/crio-42f8ed8cd4fbb726d17c88abc964b890ca8545c262cbbb5e5d02aa344eb33635 WatchSource:0}: Error finding container 42f8ed8cd4fbb726d17c88abc964b890ca8545c262cbbb5e5d02aa344eb33635: Status 404 returned error can't find the container with id 42f8ed8cd4fbb726d17c88abc964b890ca8545c262cbbb5e5d02aa344eb33635
Nov 01 22:33:24.357100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357080    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.357190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357132    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.357190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357181    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.357348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357207    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9gp\" (UniqueName: \"kubernetes.io/projected/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-kube-api-access-hs9gp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.357348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357239    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.357348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357284    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.357625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357578    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.357625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357597    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.357750 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.357644    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.359365 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.359346    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.359711 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.359693    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.411883 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.411829    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9gp\" (UniqueName: \"kubernetes.io/projected/22a9dcd7-fec2-4d08-93c9-09ff3ec499a0-kube-api-access-hs9gp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v\" (UID: \"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.455830 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.455805    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:24.832076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.832033    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"]
Nov 01 22:33:24.834597 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:33:24.834571    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a9dcd7_fec2_4d08_93c9_09ff3ec499a0.slice/crio-95bd079a3f6d7f201bcd02447c145678821f000ce1cf5230939c2c70decab2bf WatchSource:0}: Error finding container 95bd079a3f6d7f201bcd02447c145678821f000ce1cf5230939c2c70decab2bf: Status 404 returned error can't find the container with id 95bd079a3f6d7f201bcd02447c145678821f000ce1cf5230939c2c70decab2bf
Nov 01 22:33:24.868275 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.868252    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v" event={"ID":"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0","Type":"ContainerStarted","Data":"95bd079a3f6d7f201bcd02447c145678821f000ce1cf5230939c2c70decab2bf"}
Nov 01 22:33:24.869463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.869441    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" event={"ID":"832a757e-f762-446b-aa18-5425bad1c560","Type":"ContainerStarted","Data":"42f8ed8cd4fbb726d17c88abc964b890ca8545c262cbbb5e5d02aa344eb33635"}
Nov 01 22:33:24.880677 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:24.880652    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-t294z"
Nov 01 22:33:25.875435 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:25.875349    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v" event={"ID":"22a9dcd7-fec2-4d08-93c9-09ff3ec499a0","Type":"ContainerStarted","Data":"034099288e51ef378429ac8b309b96b394fa2b6c0c7c56eb42d721e50e394190"}
Nov 01 22:33:25.985278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:25.985224    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v" podStartSLOduration=1.3512497190000001 podStartE2EDuration="1.985207064s" totalImagesPullingTime="633.957345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:25.984850574 +0000 UTC m=+587.574049634" watchObservedRunningTime="2026-11-01 22:33:25.985207064 +0000 UTC m=+587.574406118"
Nov 01 22:33:26.456199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:26.456166    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:30.219207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:30.219177    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:33:30.219490 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:30.219243    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:33:30.219490 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:30.219271    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:33:30.899145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:30.899110    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" event={"ID":"832a757e-f762-446b-aa18-5425bad1c560","Type":"ContainerStarted","Data":"12faf3d05567e3cdd100455242986f36ae3b04dd85ba72a306d92ebb66b3eae0"}
Nov 01 22:33:31.057634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:31.057582    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podStartSLOduration=2.195684071 podStartE2EDuration="8.057569197s" totalImagesPullingTime="5.861885126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:31.057314622 +0000 UTC m=+592.646513685" watchObservedRunningTime="2026-11-01 22:33:31.057569197 +0000 UTC m=+592.646768257"
Nov 01 22:33:31.892760 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:31.892728    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:31.894101 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:31.894082    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:33:31.894179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:31.894126    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 22:33:32.892657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:32.892621    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:33:32.892831 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:32.892671    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 22:33:33.892586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:33.892552    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:33:33.892749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:33.892605    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 22:33:34.456976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:34.456943    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:34.469443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:34.469419    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:34.892868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:34.892779    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:33:34.892868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:34.892837    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 22:33:34.927808 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:34.927782    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v"
Nov 01 22:33:35.892580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:35.892550    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:33:35.892964 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:35.892604    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 22:33:36.892298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:36.892259    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:33:36.892457 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:36.892351    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 22:33:38.566363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.566327    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:33:38.575687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.575665    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:33:38.723354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.723323    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"]
Nov 01 22:33:38.729625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.729605    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804096 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804058    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804413 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804376    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804554 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804429    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804559    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804599    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804626    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804809 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804685    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v92h\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-kube-api-access-4v92h\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804871 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804808    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.804935 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.804877    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.827437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.827368    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"]
Nov 01 22:33:38.892079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.892022    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:38.905941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.905914    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.905980    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906011    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906095    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906127    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906149    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906201    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v92h\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-kube-api-access-4v92h\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906244    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906272    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906455 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906396    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906515    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906762    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.906874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.906856    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.907298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.907275    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.908845 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.908818    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.908962 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.908941    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.933197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.933163    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:33:38.933297 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.933219    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:33:38.959542 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.959511    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v92h\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-kube-api-access-4v92h\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.959845 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.959827    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pnpf\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:38.962939 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:38.962917    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:33:39.041932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.041905    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:39.461477 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.461443    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"]
Nov 01 22:33:39.465413 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:33:39.465381    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551392b2_5288_4eda_8e42_3bf42fb8b7d2.slice/crio-7e505ae732a6674e9194af4135657490bcfcd67a53e7ada2c02e87902da2a4dc WatchSource:0}: Error finding container 7e505ae732a6674e9194af4135657490bcfcd67a53e7ada2c02e87902da2a4dc: Status 404 returned error can't find the container with id 7e505ae732a6674e9194af4135657490bcfcd67a53e7ada2c02e87902da2a4dc
Nov 01 22:33:39.467675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.467640    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:33:39.467775 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.467697    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:33:39.467775 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.467727    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:33:39.943842 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.943798    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" event={"ID":"551392b2-5288-4eda-8e42-3bf42fb8b7d2","Type":"ContainerStarted","Data":"4868427d35c196e4cc590e1ca72fbdaa0772ab578f815b9368581c11191aa431"}
Nov 01 22:33:39.944395 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.943877    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" event={"ID":"551392b2-5288-4eda-8e42-3bf42fb8b7d2","Type":"ContainerStarted","Data":"7e505ae732a6674e9194af4135657490bcfcd67a53e7ada2c02e87902da2a4dc"}
Nov 01 22:33:39.974245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.974192    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:33:39.974416 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:39.974260    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:33:40.044202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:40.043937    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:40.044202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:40.044086    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:33:40.044202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:40.044133    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 22:33:40.062250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:40.060355    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:40.063065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:40.063016    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:33:40.175692 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:40.175632    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podStartSLOduration=2.175617755 podStartE2EDuration="2.175617755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:40.060117528 +0000 UTC m=+601.649316590" watchObservedRunningTime="2026-11-01 22:33:40.175617755 +0000 UTC m=+601.764816813"
Nov 01 22:33:41.051067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:41.049380    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:33:41.051067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:41.049438    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 22:33:42.052066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:42.051392    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:33:42.052066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:42.051461    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 22:33:43.045401 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:43.045365    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:33:43.045730 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:43.045690    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 22:33:44.045443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:44.045406    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:33:44.046066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:44.046014    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 22:33:45.049066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:45.048362    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:33:45.049066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:45.048421    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 22:33:46.042912 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:46.042880    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:33:46.043084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:46.042931    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 22:33:47.650525 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.650495    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:47.650962 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.650816    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:47.651743 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.651721    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:33:47.702563 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.702538    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-mzkxd"]
Nov 01 22:33:47.709404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.709384    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:47.758919 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.758899    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 01 22:33:47.760537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.760516    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-mzkxd"]
Nov 01 22:33:47.761417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.761400    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-nzhd9\""
Nov 01 22:33:47.761608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.761589    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 01 22:33:47.806033 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.806008    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b509139-f665-496d-911d-c97c60053ebd-metrics-certs\") pod \"ai-gateway-controller-655fc879db-mzkxd\" (UID: \"3b509139-f665-496d-911d-c97c60053ebd\") " pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:47.806180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.806113    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtpfz\" (UniqueName: \"kubernetes.io/projected/3b509139-f665-496d-911d-c97c60053ebd-kube-api-access-xtpfz\") pod \"ai-gateway-controller-655fc879db-mzkxd\" (UID: \"3b509139-f665-496d-911d-c97c60053ebd\") " pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:47.907024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.906940    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b509139-f665-496d-911d-c97c60053ebd-metrics-certs\") pod \"ai-gateway-controller-655fc879db-mzkxd\" (UID: \"3b509139-f665-496d-911d-c97c60053ebd\") " pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:47.907024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.907019    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtpfz\" (UniqueName: \"kubernetes.io/projected/3b509139-f665-496d-911d-c97c60053ebd-kube-api-access-xtpfz\") pod \"ai-gateway-controller-655fc879db-mzkxd\" (UID: \"3b509139-f665-496d-911d-c97c60053ebd\") " pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:47.909339 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.909322    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b509139-f665-496d-911d-c97c60053ebd-metrics-certs\") pod \"ai-gateway-controller-655fc879db-mzkxd\" (UID: \"3b509139-f665-496d-911d-c97c60053ebd\") " pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:47.965864 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:47.965837    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtpfz\" (UniqueName: \"kubernetes.io/projected/3b509139-f665-496d-911d-c97c60053ebd-kube-api-access-xtpfz\") pod \"ai-gateway-controller-655fc879db-mzkxd\" (UID: \"3b509139-f665-496d-911d-c97c60053ebd\") " pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:48.022509 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:48.022476    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:48.390700 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:48.390656    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-mzkxd"]
Nov 01 22:33:48.393209 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:33:48.393181    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b509139_f665_496d_911d_c97c60053ebd.slice/crio-0650fe5655c8a2a648b636c21eb703e01706aa1df4683eb6a33fcf149b1c4e6c WatchSource:0}: Error finding container 0650fe5655c8a2a648b636c21eb703e01706aa1df4683eb6a33fcf149b1c4e6c: Status 404 returned error can't find the container with id 0650fe5655c8a2a648b636c21eb703e01706aa1df4683eb6a33fcf149b1c4e6c
Nov 01 22:33:48.811126 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:33:48.811098    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:33:48.994278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:48.994241    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd" event={"ID":"3b509139-f665-496d-911d-c97c60053ebd","Type":"ContainerStarted","Data":"0650fe5655c8a2a648b636c21eb703e01706aa1df4683eb6a33fcf149b1c4e6c"}
Nov 01 22:33:56.029232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:56.029181    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd" event={"ID":"3b509139-f665-496d-911d-c97c60053ebd","Type":"ContainerStarted","Data":"a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9"}
Nov 01 22:33:56.029709 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:56.029330    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:33:56.143454 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:56.143397    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd" podStartSLOduration=2.403358896 podStartE2EDuration="9.14338484s" totalImagesPullingTime="6.740025944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:56.142222273 +0000 UTC m=+617.731421335" watchObservedRunningTime="2026-11-01 22:33:56.14338484 +0000 UTC m=+617.732583900"
Nov 01 22:33:58.291003 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.290974    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"]
Nov 01 22:33:58.294663 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.294643    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:33:58.390020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.389979    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 22:33:58.390020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.389979    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bqksj\""
Nov 01 22:33:58.392534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.392496    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"]
Nov 01 22:33:58.394525 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.394494    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a7115a-269b-499d-8dd7-5e92d093bb8e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-5h4j6\" (UID: \"58a7115a-269b-499d-8dd7-5e92d093bb8e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:33:58.394643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.394599    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2m9j\" (UniqueName: \"kubernetes.io/projected/58a7115a-269b-499d-8dd7-5e92d093bb8e-kube-api-access-g2m9j\") pod \"payload-pre-processing-7b98bb678c-5h4j6\" (UID: \"58a7115a-269b-499d-8dd7-5e92d093bb8e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:33:58.437949 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.437913    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"]
Nov 01 22:33:58.441520 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.441488    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:33:58.493434 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.493388    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"]
Nov 01 22:33:58.495232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.495201    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8cad21-8361-4f7e-b87e-17359415a82e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-xzwhj\" (UID: \"2c8cad21-8361-4f7e-b87e-17359415a82e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:33:58.495373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.495244    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z767m\" (UniqueName: \"kubernetes.io/projected/2c8cad21-8361-4f7e-b87e-17359415a82e-kube-api-access-z767m\") pod \"payload-processing-55fbbbf9dc-xzwhj\" (UID: \"2c8cad21-8361-4f7e-b87e-17359415a82e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:33:58.495446 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.495416    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a7115a-269b-499d-8dd7-5e92d093bb8e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-5h4j6\" (UID: \"58a7115a-269b-499d-8dd7-5e92d093bb8e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:33:58.495508 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.495444    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2m9j\" (UniqueName: \"kubernetes.io/projected/58a7115a-269b-499d-8dd7-5e92d093bb8e-kube-api-access-g2m9j\") pod \"payload-pre-processing-7b98bb678c-5h4j6\" (UID: \"58a7115a-269b-499d-8dd7-5e92d093bb8e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:33:58.496202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.496182    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a7115a-269b-499d-8dd7-5e92d093bb8e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-5h4j6\" (UID: \"58a7115a-269b-499d-8dd7-5e92d093bb8e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:33:58.552639 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.552560    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2m9j\" (UniqueName: \"kubernetes.io/projected/58a7115a-269b-499d-8dd7-5e92d093bb8e-kube-api-access-g2m9j\") pod \"payload-pre-processing-7b98bb678c-5h4j6\" (UID: \"58a7115a-269b-499d-8dd7-5e92d093bb8e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:33:58.596703 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.596669    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8cad21-8361-4f7e-b87e-17359415a82e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-xzwhj\" (UID: \"2c8cad21-8361-4f7e-b87e-17359415a82e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:33:58.596703 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.596699    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z767m\" (UniqueName: \"kubernetes.io/projected/2c8cad21-8361-4f7e-b87e-17359415a82e-kube-api-access-z767m\") pod \"payload-processing-55fbbbf9dc-xzwhj\" (UID: \"2c8cad21-8361-4f7e-b87e-17359415a82e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:33:58.597653 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.597626    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8cad21-8361-4f7e-b87e-17359415a82e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-xzwhj\" (UID: \"2c8cad21-8361-4f7e-b87e-17359415a82e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:33:58.607173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.607149    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:33:58.651095 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.650635    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z767m\" (UniqueName: \"kubernetes.io/projected/2c8cad21-8361-4f7e-b87e-17359415a82e-kube-api-access-z767m\") pod \"payload-processing-55fbbbf9dc-xzwhj\" (UID: \"2c8cad21-8361-4f7e-b87e-17359415a82e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:33:58.757707 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.757681    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:33:58.974807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:58.974784    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"]
Nov 01 22:33:58.976387 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:33:58.976353    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a7115a_269b_499d_8dd7_5e92d093bb8e.slice/crio-a252b15ff2833a16eafc68c8a3a542fe856454b3c622c384305758b9f787f794 WatchSource:0}: Error finding container a252b15ff2833a16eafc68c8a3a542fe856454b3c622c384305758b9f787f794: Status 404 returned error can't find the container with id a252b15ff2833a16eafc68c8a3a542fe856454b3c622c384305758b9f787f794
Nov 01 22:33:59.043857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:59.043828    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6" event={"ID":"58a7115a-269b-499d-8dd7-5e92d093bb8e","Type":"ContainerStarted","Data":"a252b15ff2833a16eafc68c8a3a542fe856454b3c622c384305758b9f787f794"}
Nov 01 22:33:59.132216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:33:59.132193    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"]
Nov 01 22:33:59.133522 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:33:59.133501    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8cad21_8361_4f7e_b87e_17359415a82e.slice/crio-cd7c3d6f5d89e0551910bd82c1667e207634b8c15a67c41a4f94034803536a88 WatchSource:0}: Error finding container cd7c3d6f5d89e0551910bd82c1667e207634b8c15a67c41a4f94034803536a88: Status 404 returned error can't find the container with id cd7c3d6f5d89e0551910bd82c1667e207634b8c15a67c41a4f94034803536a88
Nov 01 22:34:00.050732 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:00.050692    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj" event={"ID":"2c8cad21-8361-4f7e-b87e-17359415a82e","Type":"ContainerStarted","Data":"cd7c3d6f5d89e0551910bd82c1667e207634b8c15a67c41a4f94034803536a88"}
Nov 01 22:34:03.067462 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:03.067422    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj" event={"ID":"2c8cad21-8361-4f7e-b87e-17359415a82e","Type":"ContainerStarted","Data":"f512647d5d7309b72ff1fe28126029e1128d783b94096f98f91b576879c1948a"}
Nov 01 22:34:03.067920 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:03.067568    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:34:03.068968 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:03.068944    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6" event={"ID":"58a7115a-269b-499d-8dd7-5e92d093bb8e","Type":"ContainerStarted","Data":"671695a2f182907b8d125f02f8bdc6350e328da5696bd254feafe83870c825b8"}
Nov 01 22:34:03.069108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:03.068998    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:34:03.175181 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:03.175136    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj" podStartSLOduration=2.189907015 podStartE2EDuration="5.175124765s" totalImagesPullingTime="2.98521775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:03.17402337 +0000 UTC m=+624.763222430" watchObservedRunningTime="2026-11-01 22:34:03.175124765 +0000 UTC m=+624.764323826"
Nov 01 22:34:03.282483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:03.282438    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6" podStartSLOduration=2.200325881 podStartE2EDuration="5.282423413s" totalImagesPullingTime="3.082097532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:03.280874078 +0000 UTC m=+624.870073139" watchObservedRunningTime="2026-11-01 22:34:03.282423413 +0000 UTC m=+624.871622475"
Nov 01 22:34:07.035428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.035401    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:34:07.393816 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.393740    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-847995f698-ll4ng"]
Nov 01 22:34:07.397266 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.397246    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:07.452225 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.452199    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-847995f698-ll4ng"]
Nov 01 22:34:07.479569 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.479543    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vfm\" (UniqueName: \"kubernetes.io/projected/7966306e-e19c-4423-a82f-5827ce9d2d63-kube-api-access-l2vfm\") pod \"ai-gateway-controller-847995f698-ll4ng\" (UID: \"7966306e-e19c-4423-a82f-5827ce9d2d63\") " pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:07.479695 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.479588    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7966306e-e19c-4423-a82f-5827ce9d2d63-metrics-certs\") pod \"ai-gateway-controller-847995f698-ll4ng\" (UID: \"7966306e-e19c-4423-a82f-5827ce9d2d63\") " pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:07.580930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.580907    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vfm\" (UniqueName: \"kubernetes.io/projected/7966306e-e19c-4423-a82f-5827ce9d2d63-kube-api-access-l2vfm\") pod \"ai-gateway-controller-847995f698-ll4ng\" (UID: \"7966306e-e19c-4423-a82f-5827ce9d2d63\") " pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:07.581080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.580957    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7966306e-e19c-4423-a82f-5827ce9d2d63-metrics-certs\") pod \"ai-gateway-controller-847995f698-ll4ng\" (UID: \"7966306e-e19c-4423-a82f-5827ce9d2d63\") " pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:07.583404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.583380    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7966306e-e19c-4423-a82f-5827ce9d2d63-metrics-certs\") pod \"ai-gateway-controller-847995f698-ll4ng\" (UID: \"7966306e-e19c-4423-a82f-5827ce9d2d63\") " pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:07.633687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.633661    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vfm\" (UniqueName: \"kubernetes.io/projected/7966306e-e19c-4423-a82f-5827ce9d2d63-kube-api-access-l2vfm\") pod \"ai-gateway-controller-847995f698-ll4ng\" (UID: \"7966306e-e19c-4423-a82f-5827ce9d2d63\") " pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:07.709633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:07.709607    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:08.070576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:08.070550    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-847995f698-ll4ng"]
Nov 01 22:34:08.072454 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:34:08.072420    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7966306e_e19c_4423_a82f_5827ce9d2d63.slice/crio-adb463827067fab65788964e8dd984fdeea643fbe4ab0f679f0c4fe6e8294f58 WatchSource:0}: Error finding container adb463827067fab65788964e8dd984fdeea643fbe4ab0f679f0c4fe6e8294f58: Status 404 returned error can't find the container with id adb463827067fab65788964e8dd984fdeea643fbe4ab0f679f0c4fe6e8294f58
Nov 01 22:34:08.092808 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:08.092778    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-847995f698-ll4ng" event={"ID":"7966306e-e19c-4423-a82f-5827ce9d2d63","Type":"ContainerStarted","Data":"adb463827067fab65788964e8dd984fdeea643fbe4ab0f679f0c4fe6e8294f58"}
Nov 01 22:34:10.105731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:10.105691    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-847995f698-ll4ng" event={"ID":"7966306e-e19c-4423-a82f-5827ce9d2d63","Type":"ContainerStarted","Data":"48e5d2d3b6f2d2348aa235c699f51c1d26030af4761c8c66fa08868ea285078f"}
Nov 01 22:34:10.106124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:10.105822    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:10.215819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:10.215774    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-847995f698-ll4ng" podStartSLOduration=2.161107952 podStartE2EDuration="3.215760888s" totalImagesPullingTime="1.054652936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:10.214970537 +0000 UTC m=+631.804169598" watchObservedRunningTime="2026-11-01 22:34:10.215760888 +0000 UTC m=+631.804959948"
Nov 01 22:34:14.076398 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:14.076367    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-xzwhj"
Nov 01 22:34:14.076782 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:14.076428    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-5h4j6"
Nov 01 22:34:18.811779 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:34:18.811746    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:34:21.113104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.113073    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-847995f698-ll4ng"
Nov 01 22:34:21.267100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.267069    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-mzkxd"]
Nov 01 22:34:21.267267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.267250    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd" podUID="3b509139-f665-496d-911d-c97c60053ebd" containerName="manager" containerID="cri-o://a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9" gracePeriod=30
Nov 01 22:34:21.547174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.547146    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:34:21.695605 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.695576    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b509139-f665-496d-911d-c97c60053ebd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b509139-f665-496d-911d-c97c60053ebd-metrics-certs\") pod \"3b509139-f665-496d-911d-c97c60053ebd\" (UID: \"3b509139-f665-496d-911d-c97c60053ebd\") "
Nov 01 22:34:21.695761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.695684    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b509139-f665-496d-911d-c97c60053ebd-kube-api-access-xtpfz\" (UniqueName: \"kubernetes.io/projected/3b509139-f665-496d-911d-c97c60053ebd-kube-api-access-xtpfz\") pod \"3b509139-f665-496d-911d-c97c60053ebd\" (UID: \"3b509139-f665-496d-911d-c97c60053ebd\") "
Nov 01 22:34:21.702883 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.702785    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b509139-f665-496d-911d-c97c60053ebd-metrics-certs" pod "3b509139-f665-496d-911d-c97c60053ebd" (UID: "3b509139-f665-496d-911d-c97c60053ebd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:34:21.704354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.704326    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b509139-f665-496d-911d-c97c60053ebd-kube-api-access-xtpfz" pod "3b509139-f665-496d-911d-c97c60053ebd" (UID: "3b509139-f665-496d-911d-c97c60053ebd"). InnerVolumeSpecName "kube-api-access-xtpfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:34:21.797052 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.797020    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b509139-f665-496d-911d-c97c60053ebd-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:21.797191 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:21.797068    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtpfz\" (UniqueName: \"kubernetes.io/projected/3b509139-f665-496d-911d-c97c60053ebd-kube-api-access-xtpfz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:22.161180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.161070    4837 generic.go:356] "Generic (PLEG): container finished" podID="3b509139-f665-496d-911d-c97c60053ebd" containerID="a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9" exitCode=0
Nov 01 22:34:22.161180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.161134    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd"
Nov 01 22:34:22.161180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.161157    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd" event={"ID":"3b509139-f665-496d-911d-c97c60053ebd","Type":"ContainerDied","Data":"a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9"}
Nov 01 22:34:22.161641 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.161230    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-655fc879db-mzkxd" event={"ID":"3b509139-f665-496d-911d-c97c60053ebd","Type":"ContainerDied","Data":"0650fe5655c8a2a648b636c21eb703e01706aa1df4683eb6a33fcf149b1c4e6c"}
Nov 01 22:34:22.161641 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.161253    4837 scope.go:122] "RemoveContainer" containerID="a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9"
Nov 01 22:34:22.169511 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.169497    4837 scope.go:122] "RemoveContainer" containerID="a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9"
Nov 01 22:34:22.169757 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:34:22.169737    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9\": container with ID starting with a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9 not found: ID does not exist" containerID="a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9"
Nov 01 22:34:22.169805 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.169766    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9"} err="failed to get container status \"a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9\": rpc error: code = NotFound desc = could not find container \"a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9\": container with ID starting with a2495c9e115c650002b5eec4cb1360be26faeabe8c702781d5d2f069b0ab84f9 not found: ID does not exist"
Nov 01 22:34:22.312895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.312866    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-mzkxd"]
Nov 01 22:34:22.359307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.359287    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-655fc879db-mzkxd"]
Nov 01 22:34:22.523620 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:22.523597    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b509139-f665-496d-911d-c97c60053ebd" path="/var/lib/kubelet/pods/3b509139-f665-496d-911d-c97c60053ebd/volumes"
Nov 01 22:34:24.080926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.080896    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"]
Nov 01 22:34:24.081321 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.081307    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3b509139-f665-496d-911d-c97c60053ebd" containerName="manager"
Nov 01 22:34:24.081380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.081323    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b509139-f665-496d-911d-c97c60053ebd" containerName="manager"
Nov 01 22:34:24.081419 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.081383    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b509139-f665-496d-911d-c97c60053ebd" containerName="manager"
Nov 01 22:34:24.089842 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.089821    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.127230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.127210    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"]
Nov 01 22:34:24.130848 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.130832    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.215212 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.215184    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"]
Nov 01 22:34:24.217658 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.217626    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.217773 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.217668    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fx42\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-kube-api-access-7fx42\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.217846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.217805    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.217907 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.217868    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.217963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.217940    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.218013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.217979    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955785c3-9b45-4da1-ae35-63861a17b4a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.218013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218007    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.218137 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218120    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.218172 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218161    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535a603e-c963-4fc8-a816-2e1b8209120e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.218218 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218196    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.218630 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218597    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.218731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218646    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.218731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218673    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.218731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218715    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzk87\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-kube-api-access-nzk87\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.218849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218820    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.218911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218860    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535a603e-c963-4fc8-a816-2e1b8209120e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.218969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218939    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.219026 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.218969    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.278394 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.278366    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"]
Nov 01 22:34:24.320383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320356    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.320524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320410    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.320524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320445    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.320524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320469    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955785c3-9b45-4da1-ae35-63861a17b4a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.320524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320494    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.320524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320522    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.320850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320552    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535a603e-c963-4fc8-a816-2e1b8209120e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.320850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320585    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.320850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320613    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.320850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320653    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.320850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320684    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.320850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320803    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzk87\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-kube-api-access-nzk87\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.320850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320819    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.320850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320843    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320873    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535a603e-c963-4fc8-a816-2e1b8209120e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320910    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320909    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320938    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320960    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320972    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.320993    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.321016    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fx42\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-kube-api-access-7fx42\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.321272    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.321324    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535a603e-c963-4fc8-a816-2e1b8209120e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.321327    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.321759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.321573    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.321759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.321615    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.322181 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.322160    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955785c3-9b45-4da1-ae35-63861a17b4a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.323146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.323123    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.323236 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.323201    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.323695 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.323673    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535a603e-c963-4fc8-a816-2e1b8209120e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.323695 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.323686    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.376834 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.376777    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.377091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.377073    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzk87\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-kube-api-access-nzk87\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rsr62\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.377277 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.377256    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.377408 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.377392    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fx42\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-kube-api-access-7fx42\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h49jz\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.403104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.403084    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:24.442863 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.442838    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:24.819442 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.819416    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"]
Nov 01 22:34:24.820861 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:34:24.820614    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535a603e_c963_4fc8_a816_2e1b8209120e.slice/crio-2f1cbd141808feca7e7b8edebe546cbbbeac4178a0f809f1301d65d950ee7f80 WatchSource:0}: Error finding container 2f1cbd141808feca7e7b8edebe546cbbbeac4178a0f809f1301d65d950ee7f80: Status 404 returned error can't find the container with id 2f1cbd141808feca7e7b8edebe546cbbbeac4178a0f809f1301d65d950ee7f80
Nov 01 22:34:24.823900 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.823877    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"]
Nov 01 22:34:24.823980 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.823920    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:24.824029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.823991    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:24.824092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.824034    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:24.825591 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:34:24.825568    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955785c3_9b45_4da1_ae35_63861a17b4a3.slice/crio-63d6c01ad72fd0ca71e02bf87957edff7b1150387bc0ad8d8ecdae9614c6fba3 WatchSource:0}: Error finding container 63d6c01ad72fd0ca71e02bf87957edff7b1150387bc0ad8d8ecdae9614c6fba3: Status 404 returned error can't find the container with id 63d6c01ad72fd0ca71e02bf87957edff7b1150387bc0ad8d8ecdae9614c6fba3
Nov 01 22:34:24.830843 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.830818    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:24.830916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.830875    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:24.830916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:24.830904    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:25.189371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.189332    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" event={"ID":"535a603e-c963-4fc8-a816-2e1b8209120e","Type":"ContainerStarted","Data":"3cdc2b41f29e9c5254be08aeb0c8c607319c4ae6eec484da6eec3647e9af3bb5"}
Nov 01 22:34:25.189835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.189431    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" event={"ID":"535a603e-c963-4fc8-a816-2e1b8209120e","Type":"ContainerStarted","Data":"2f1cbd141808feca7e7b8edebe546cbbbeac4178a0f809f1301d65d950ee7f80"}
Nov 01 22:34:25.190989 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.190961    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" event={"ID":"955785c3-9b45-4da1-ae35-63861a17b4a3","Type":"ContainerStarted","Data":"054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0"}
Nov 01 22:34:25.191136 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.191011    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" event={"ID":"955785c3-9b45-4da1-ae35-63861a17b4a3","Type":"ContainerStarted","Data":"63d6c01ad72fd0ca71e02bf87957edff7b1150387bc0ad8d8ecdae9614c6fba3"}
Nov 01 22:34:25.303751 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.303696    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podStartSLOduration=1.303681308 podStartE2EDuration="1.303681308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:25.302574357 +0000 UTC m=+646.891773443" watchObservedRunningTime="2026-11-01 22:34:25.303681308 +0000 UTC m=+646.892880371"
Nov 01 22:34:25.403364 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.403328    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:25.404972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.404942    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h49jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 22:34:25.405112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.404999    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 22:34:25.449122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.448430    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:25.470066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.457175    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 22:34:25.470066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.457239    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 22:34:25.470066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:25.458533    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podStartSLOduration=1.458514394 podStartE2EDuration="1.458514394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:25.4577471 +0000 UTC m=+647.046946162" watchObservedRunningTime="2026-11-01 22:34:25.458514394 +0000 UTC m=+647.047713456"
Nov 01 22:34:26.403787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:26.403755    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h49jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 22:34:26.404208 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:26.403810    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 22:34:26.446069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:26.444412    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 22:34:26.446069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:26.444476    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 22:34:27.404279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:27.404244    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h49jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 22:34:27.404669 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:27.404299    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 22:34:27.443505 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:27.443462    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 22:34:27.443784 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:27.443755    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 22:34:28.404128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:28.404099    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h49jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 22:34:28.404353 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:28.404332    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 22:34:28.450171 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:28.450138    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 22:34:28.450366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:28.450340    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 22:34:29.403582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:29.403539    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h49jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 22:34:29.403742 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:29.403602    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 22:34:29.445066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:29.443466    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 22:34:29.445066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:29.443563    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 22:34:30.403945 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:30.403909    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h49jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 22:34:30.404159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:30.403994    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 22:34:30.448510 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:30.448470    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 22:34:30.449133 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:30.449099    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 22:34:31.403820 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:31.403779    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h49jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 22:34:31.404013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:31.403907    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 22:34:31.443503 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:31.443466    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 22:34:31.443636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:31.443518    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 22:34:32.443610 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:32.443568    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 22:34:32.443983 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:32.443637    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 22:34:33.222450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:33.222362    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:33.223076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:33.222996    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:33.223869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:33.223846    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:34:33.447298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:33.447271    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:33.447672 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:33.447561    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:33.448257 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:33.448242    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:34:48.814082 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:34:48.813984    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:34:53.837763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.837729    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"]
Nov 01 22:34:53.841354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.841336    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.900382 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.900350    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e71d55-openshift-default-dockercfg-696c5\""
Nov 01 22:34:53.918869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.918830    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.919110 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.919087    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.919245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.919139    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.919312 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.919290    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.919370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.919343    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.919480 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.919456    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8p4\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-kube-api-access-hx8p4\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.919536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.919505    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cddca7c6-47ca-440d-847e-1f1178de7051-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.919611 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.919595    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cddca7c6-47ca-440d-847e-1f1178de7051-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.919675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.919657    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:53.942339 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:53.942317    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"]
Nov 01 22:34:54.021215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021179    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021228    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021274    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8p4\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-kube-api-access-hx8p4\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021303    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cddca7c6-47ca-440d-847e-1f1178de7051-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021359    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cddca7c6-47ca-440d-847e-1f1178de7051-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021426    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021454    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021482    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021537    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021709    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021767    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.021926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021868    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.022009 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.021922    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.022131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.022110    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cddca7c6-47ca-440d-847e-1f1178de7051-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.023788 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.023763    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.023988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.023969    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cddca7c6-47ca-440d-847e-1f1178de7051-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.085854 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.085823    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.086123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.086104    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8p4\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-kube-api-access-hx8p4\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.161168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.161103    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:54.199483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.199452    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"]
Nov 01 22:34:54.203368 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.203348    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.273301 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.273247    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c54de0-openshift-default-dockercfg-k2kn2\""
Nov 01 22:34:54.275406 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.275374    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"]
Nov 01 22:34:54.334227 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334200    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b5a76c-198e-4891-8383-a92528dcc6ca-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.334374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334245    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.334374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334274    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mvc\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-kube-api-access-c5mvc\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.334462 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334399    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.334462 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334426    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.334553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334502    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.334590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334561    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.334632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334589    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.334729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.334708    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444091    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444465    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444563    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444597    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444618    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b5a76c-198e-4891-8383-a92528dcc6ca-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444831 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444697    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444831 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444741    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mvc\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-kube-api-access-c5mvc\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444831 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444776    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.444831 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444802    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.445072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.444851    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.445283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.445261    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.445406 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.445286    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b5a76c-198e-4891-8383-a92528dcc6ca-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.445925 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.445521    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.445925 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.445571    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.451170 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.451138    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.453285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.453241    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.507564 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.507531    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mvc\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-kube-api-access-c5mvc\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.511957 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.511924    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.530574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.530547    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:54.579491 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.579464    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"]
Nov 01 22:34:54.581066 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:34:54.580923    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddca7c6_47ca_440d_847e_1f1178de7051.slice/crio-a1ad3b50ae0a3fdf88c379a60ad9cc8d52f7f6f0110e7f3d4142fb7c36801725 WatchSource:0}: Error finding container a1ad3b50ae0a3fdf88c379a60ad9cc8d52f7f6f0110e7f3d4142fb7c36801725: Status 404 returned error can't find the container with id a1ad3b50ae0a3fdf88c379a60ad9cc8d52f7f6f0110e7f3d4142fb7c36801725
Nov 01 22:34:54.584187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.584143    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:54.584287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.584235    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:54.584349 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.584285    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:54.924912 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.924810    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"]
Nov 01 22:34:54.939897 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:34:54.937430    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b5a76c_198e_4891_8383_a92528dcc6ca.slice/crio-435b21988025a06a07e805a54bcb5b26903ca17e0735ce757288b096a68167fc WatchSource:0}: Error finding container 435b21988025a06a07e805a54bcb5b26903ca17e0735ce757288b096a68167fc: Status 404 returned error can't find the container with id 435b21988025a06a07e805a54bcb5b26903ca17e0735ce757288b096a68167fc
Nov 01 22:34:54.944130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.944086    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:54.944249 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.944173    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:54.944318 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:54.944264    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:55.190421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.190342    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"]
Nov 01 22:34:55.200303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.200271    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.252890    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"]
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.253263    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9dfe9f-openshift-default-dockercfg-9r4dj\""
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254571    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254616    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254659    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254724    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254760    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254783    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254852    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562f9bc-b7b9-4885-9050-35e6200c19a7-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254886    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.255982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.254913    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2chs\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-kube-api-access-l2chs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.334072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.334028    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" event={"ID":"cddca7c6-47ca-440d-847e-1f1178de7051","Type":"ContainerStarted","Data":"65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9"}
Nov 01 22:34:55.334323 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.334296    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" event={"ID":"cddca7c6-47ca-440d-847e-1f1178de7051","Type":"ContainerStarted","Data":"a1ad3b50ae0a3fdf88c379a60ad9cc8d52f7f6f0110e7f3d4142fb7c36801725"}
Nov 01 22:34:55.335760 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.335725    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" event={"ID":"85b5a76c-198e-4891-8383-a92528dcc6ca","Type":"ContainerStarted","Data":"aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d"}
Nov 01 22:34:55.335872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.335783    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" event={"ID":"85b5a76c-198e-4891-8383-a92528dcc6ca","Type":"ContainerStarted","Data":"435b21988025a06a07e805a54bcb5b26903ca17e0735ce757288b096a68167fc"}
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359228    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359305    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359383    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359419    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359443    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359512    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562f9bc-b7b9-4885-9050-35e6200c19a7-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359554    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359585    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2chs\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-kube-api-access-l2chs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.361504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.359660    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.362553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.362523    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.364446 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.364419    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.381643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.364595    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.381643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.365247    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562f9bc-b7b9-4885-9050-35e6200c19a7-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.381643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.365584    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.381643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.367888    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.381643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.368838    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.418054 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.417998    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.418849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.418821    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2chs\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-kube-api-access-l2chs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.456074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.453354    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" podStartSLOduration=2.45333855 podStartE2EDuration="2.45333855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:55.451428951 +0000 UTC m=+677.040628412" watchObservedRunningTime="2026-11-01 22:34:55.45333855 +0000 UTC m=+677.042537617"
Nov 01 22:34:55.516382 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.516353    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:55.531492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.531466    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:55.566116 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.565360    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podStartSLOduration=1.565342005 podStartE2EDuration="1.565342005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:55.563070009 +0000 UTC m=+677.152269093" watchObservedRunningTime="2026-11-01 22:34:55.565342005 +0000 UTC m=+677.154541071"
Nov 01 22:34:55.939368 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.939313    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"]
Nov 01 22:34:55.941033 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:34:55.941005    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0562f9bc_b7b9_4885_9050_35e6200c19a7.slice/crio-701595d09e98d44ca8a9818c147f2666490153776d822a0d0bbf260c9f409cac WatchSource:0}: Error finding container 701595d09e98d44ca8a9818c147f2666490153776d822a0d0bbf260c9f409cac: Status 404 returned error can't find the container with id 701595d09e98d44ca8a9818c147f2666490153776d822a0d0bbf260c9f409cac
Nov 01 22:34:55.943276 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.943247    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:55.943352 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.943308    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:55.943352 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:55.943336    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:34:56.161637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.161604    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:56.166345 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.166320    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:56.346450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.346347    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" event={"ID":"0562f9bc-b7b9-4885-9050-35e6200c19a7","Type":"ContainerStarted","Data":"f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a"}
Nov 01 22:34:56.346450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.346418    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" event={"ID":"0562f9bc-b7b9-4885-9050-35e6200c19a7","Type":"ContainerStarted","Data":"701595d09e98d44ca8a9818c147f2666490153776d822a0d0bbf260c9f409cac"}
Nov 01 22:34:56.347009 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.346984    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:56.348002 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.347982    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:34:56.482386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.482320    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" podStartSLOduration=1.482302713 podStartE2EDuration="1.482302713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:56.482199114 +0000 UTC m=+678.071398177" watchObservedRunningTime="2026-11-01 22:34:56.482302713 +0000 UTC m=+678.071501775"
Nov 01 22:34:56.525188 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.525160    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:56.525361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.525253    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:56.535616 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:56.535590    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:57.351736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:57.351702    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:57.352166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:57.351756    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:34:57.352380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:57.352363    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:34:57.352425 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:34:57.352412    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:35:11.752214 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:11.752178    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"]
Nov 01 22:35:11.761285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:11.761260    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:11.816304 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:11.816276    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"]
Nov 01 22:35:11.859723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:11.859700    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-e71d55\""
Nov 01 22:35:11.859723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:11.859701    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-e71d55-dockercfg-nvpdr\""
Nov 01 22:35:11.953527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:11.953503    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-kube-api-access-z7vnk\") pod \"payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d\" (UID: \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:11.953672 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:11.953533    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d\" (UID: \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:12.054744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:12.054654    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-kube-api-access-z7vnk\") pod \"payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d\" (UID: \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:12.054982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:12.054962    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d\" (UID: \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:12.055975 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:12.055947    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d\" (UID: \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:12.123903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:12.123870    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-kube-api-access-z7vnk\") pod \"payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d\" (UID: \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:12.300065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:12.297786    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:35:12.374742 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:12.374666    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:12.743784 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:12.743760    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"]
Nov 01 22:35:12.745742 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:35:12.745715    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ac5f8f_d582_4e2a_b94b_2f727121a5f7.slice/crio-aec140513228dc823c2ec784168f87797b7d21e7ad6befd0d2540d63cb39a293 WatchSource:0}: Error finding container aec140513228dc823c2ec784168f87797b7d21e7ad6befd0d2540d63cb39a293: Status 404 returned error can't find the container with id aec140513228dc823c2ec784168f87797b7d21e7ad6befd0d2540d63cb39a293
Nov 01 22:35:13.455623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:13.455586    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d" event={"ID":"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7","Type":"ContainerStarted","Data":"db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c"}
Nov 01 22:35:13.456064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:13.455647    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d" event={"ID":"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7","Type":"ContainerStarted","Data":"aec140513228dc823c2ec784168f87797b7d21e7ad6befd0d2540d63cb39a293"}
Nov 01 22:35:13.456064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:13.455731    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:13.566067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:13.566007    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d" podStartSLOduration=2.5659929630000002 podStartE2EDuration="2.565992963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:13.564157571 +0000 UTC m=+695.153356651" watchObservedRunningTime="2026-11-01 22:35:13.565992963 +0000 UTC m=+695.155192023"
Nov 01 22:35:14.254397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.254358    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"]
Nov 01 22:35:14.258541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.258519    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:14.286165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.286141    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cef0b8-cc9d-4821-9456-e3189cf59ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw\" (UID: \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:14.286271 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.286196    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wls\" (UniqueName: \"kubernetes.io/projected/76cef0b8-cc9d-4821-9456-e3189cf59ec5-kube-api-access-86wls\") pod \"payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw\" (UID: \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:14.308257 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.308228    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c54de0\""
Nov 01 22:35:14.308358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.308349    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c54de0-dockercfg-866p9\""
Nov 01 22:35:14.396166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.394018    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cef0b8-cc9d-4821-9456-e3189cf59ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw\" (UID: \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:14.396643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.396613    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wls\" (UniqueName: \"kubernetes.io/projected/76cef0b8-cc9d-4821-9456-e3189cf59ec5-kube-api-access-86wls\") pod \"payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw\" (UID: \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:14.397757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.397690    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"]
Nov 01 22:35:14.397954 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.397924    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cef0b8-cc9d-4821-9456-e3189cf59ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw\" (UID: \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:14.403834 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.403558    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"]
Nov 01 22:35:14.408760 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.408742    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:14.443098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.443036    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"]
Nov 01 22:35:14.450861 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.450837    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wls\" (UniqueName: \"kubernetes.io/projected/76cef0b8-cc9d-4821-9456-e3189cf59ec5-kube-api-access-86wls\") pod \"payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw\" (UID: \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:14.497722 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.497695    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49g4v\" (UniqueName: \"kubernetes.io/projected/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-kube-api-access-49g4v\") pod \"payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw\" (UID: \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:14.498084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.497854    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw\" (UID: \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:14.571226 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.571166    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:14.599328 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.599301    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw\" (UID: \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:14.599458 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.599381    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49g4v\" (UniqueName: \"kubernetes.io/projected/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-kube-api-access-49g4v\") pod \"payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw\" (UID: \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:14.599974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.599948    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw\" (UID: \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:14.653905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.653882    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49g4v\" (UniqueName: \"kubernetes.io/projected/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-kube-api-access-49g4v\") pod \"payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw\" (UID: \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:14.721443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.721414    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:14.938895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:14.938864    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"]
Nov 01 22:35:14.939476 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:35:14.939444    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cef0b8_cc9d_4821_9456_e3189cf59ec5.slice/crio-eb9bdea4457e2fc135ac594ec08702dae69ecc3a93f3387c8aa89960a9e203ac WatchSource:0}: Error finding container eb9bdea4457e2fc135ac594ec08702dae69ecc3a93f3387c8aa89960a9e203ac: Status 404 returned error can't find the container with id eb9bdea4457e2fc135ac594ec08702dae69ecc3a93f3387c8aa89960a9e203ac
Nov 01 22:35:15.085613 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.085590    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"]
Nov 01 22:35:15.086975 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:35:15.086943    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f99ea57_d150_4f65_b5e3_6c1ea645fd1c.slice/crio-f565711648aa8e7b1f544a2f4b636a2a3e9c9fde5fb0d284abf4ffc458df0fca WatchSource:0}: Error finding container f565711648aa8e7b1f544a2f4b636a2a3e9c9fde5fb0d284abf4ffc458df0fca: Status 404 returned error can't find the container with id f565711648aa8e7b1f544a2f4b636a2a3e9c9fde5fb0d284abf4ffc458df0fca
Nov 01 22:35:15.466977 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.466949    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw" event={"ID":"76cef0b8-cc9d-4821-9456-e3189cf59ec5","Type":"ContainerStarted","Data":"c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9"}
Nov 01 22:35:15.467178 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.467007    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw" event={"ID":"76cef0b8-cc9d-4821-9456-e3189cf59ec5","Type":"ContainerStarted","Data":"eb9bdea4457e2fc135ac594ec08702dae69ecc3a93f3387c8aa89960a9e203ac"}
Nov 01 22:35:15.467178 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.467034    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:15.468488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.468463    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw" event={"ID":"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c","Type":"ContainerStarted","Data":"9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147"}
Nov 01 22:35:15.468605 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.468503    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw" event={"ID":"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c","Type":"ContainerStarted","Data":"f565711648aa8e7b1f544a2f4b636a2a3e9c9fde5fb0d284abf4ffc458df0fca"}
Nov 01 22:35:15.468605 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.468523    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:15.576037 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.575986    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw" podStartSLOduration=1.575973815 podStartE2EDuration="1.575973815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:15.574457464 +0000 UTC m=+697.163656525" watchObservedRunningTime="2026-11-01 22:35:15.575973815 +0000 UTC m=+697.165172894"
Nov 01 22:35:15.698248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:15.687233    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw" podStartSLOduration=1.6872169970000002 podStartE2EDuration="1.687216997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:15.684344726 +0000 UTC m=+697.273543804" watchObservedRunningTime="2026-11-01 22:35:15.687216997 +0000 UTC m=+697.276416059"
Nov 01 22:35:18.814455 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:35:18.814421    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:35:21.148137 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:21.148095    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:35:21.976782 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:21.968474    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"]
Nov 01 22:35:21.981571 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:21.981546    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:22.007057 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.006915    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"]
Nov 01 22:35:22.054067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.051276    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a6dd2f\""
Nov 01 22:35:22.054067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.051468    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a6dd2f-dockercfg-bpjhq\""
Nov 01 22:35:22.093119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.093086    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m\" (UID: \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:22.093264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.093215    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqks\" (UniqueName: \"kubernetes.io/projected/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-kube-api-access-mgqks\") pod \"payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m\" (UID: \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:22.193904 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.193870    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqks\" (UniqueName: \"kubernetes.io/projected/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-kube-api-access-mgqks\") pod \"payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m\" (UID: \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:22.194361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.193960    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m\" (UID: \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:22.194820 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.194791    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m\" (UID: \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:22.262248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.262159    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqks\" (UniqueName: \"kubernetes.io/projected/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-kube-api-access-mgqks\") pod \"payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m\" (UID: \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:22.294891 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.294851    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:22.691657 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:35:22.691630    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afb93f5_fcee_4434_bbfd_e297f8ea86cb.slice/crio-c0007e381682628a86f10e0ebf1fd103b9b74b238ced4e5b633c04eab90d68ce WatchSource:0}: Error finding container c0007e381682628a86f10e0ebf1fd103b9b74b238ced4e5b633c04eab90d68ce: Status 404 returned error can't find the container with id c0007e381682628a86f10e0ebf1fd103b9b74b238ced4e5b633c04eab90d68ce
Nov 01 22:35:22.692389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:22.692364    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"]
Nov 01 22:35:23.568738 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:23.568707    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m" event={"ID":"7afb93f5-fcee-4434-bbfd-e297f8ea86cb","Type":"ContainerStarted","Data":"ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c"}
Nov 01 22:35:23.569139 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:23.568757    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m" event={"ID":"7afb93f5-fcee-4434-bbfd-e297f8ea86cb","Type":"ContainerStarted","Data":"c0007e381682628a86f10e0ebf1fd103b9b74b238ced4e5b633c04eab90d68ce"}
Nov 01 22:35:23.569139 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:23.568836    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:23.680417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:23.680367    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m" podStartSLOduration=2.680352128 podStartE2EDuration="2.680352128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:23.67932793 +0000 UTC m=+705.268526991" watchObservedRunningTime="2026-11-01 22:35:23.680352128 +0000 UTC m=+705.269551242"
Nov 01 22:35:24.463846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:24.463823    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 22:35:26.495080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:26.492851    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 22:35:26.495080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:26.493395    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 22:35:29.884649 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:29.884612    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:35:30.615141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:30.605774    4837 generic.go:356] "Generic (PLEG): container finished" podID="cddca7c6-47ca-440d-847e-1f1178de7051" containerID="65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9" exitCode=0
Nov 01 22:35:30.615141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:30.605845    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" event={"ID":"cddca7c6-47ca-440d-847e-1f1178de7051","Type":"ContainerDied","Data":"65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9"}
Nov 01 22:35:30.615141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:30.606306    4837 scope.go:122] "RemoveContainer" containerID="65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9"
Nov 01 22:35:30.615141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:30.608761    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:35:30.615141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:30.608864    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:35:30.615141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:30.608940    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:35:31.652066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:31.645579    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" event={"ID":"cddca7c6-47ca-440d-847e-1f1178de7051","Type":"ContainerStarted","Data":"0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09"}
Nov 01 22:35:32.161718 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:32.161685    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:35:32.161895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:32.161851    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 22:35:32.161941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:32.161894    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 22:35:33.161513 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:33.161472    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 22:35:33.161900 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:33.161535    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 22:35:34.162124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:34.162083    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 22:35:34.162608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:34.162160    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 22:35:34.577351 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:34.577325    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 22:35:35.161849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:35.161804    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 22:35:35.162087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:35.161884    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 22:35:36.153603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.153569    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"]
Nov 01 22:35:36.159808 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.159785    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.162310 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.162283    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 22:35:36.162411 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.162350    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 22:35:36.255445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.255409    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"kube-root-ca.crt\""
Nov 01 22:35:36.255445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.255454    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"openshift-service-ca.crt\""
Nov 01 22:35:36.255708 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.255559    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"default-dockercfg-72fgz\""
Nov 01 22:35:36.255708 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.255634    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"simulator-w1-5b8fd2-kserve-self-signed-certs\""
Nov 01 22:35:36.257565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.257543    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"]
Nov 01 22:35:36.272853 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.272832    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-dshm\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.272956 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.272880    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tmp-dir\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.272956 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.272922    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwn4\" (UniqueName: \"kubernetes.io/projected/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-kube-api-access-hcwn4\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.273099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.273083    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-home\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.273141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.273115    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tls-certs\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.273174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.273142    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-model-cache\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374152    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tmp-dir\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374193    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwn4\" (UniqueName: \"kubernetes.io/projected/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-kube-api-access-hcwn4\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374250    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-home\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374269    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tls-certs\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374291    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-model-cache\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374359    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-dshm\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374650 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374625    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tmp-dir\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374712 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374629    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-home\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.374768 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.374706    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-model-cache\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.376625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.376592    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-dshm\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.376748 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.376732    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tls-certs\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.455713 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.455676    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwn4\" (UniqueName: \"kubernetes.io/projected/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-kube-api-access-hcwn4\") pod \"simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") " pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.480396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.480371    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:36.857909 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:36.857885    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"]
Nov 01 22:35:36.859602 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:35:36.859571    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c7d43c7_c0d4_47a5_80c9_6b1bcc91345b.slice/crio-0fbc1ae183c4997441cc4b32259800a352ffffad3a05db4ff9d0c3aeacd5a33a WatchSource:0}: Error finding container 0fbc1ae183c4997441cc4b32259800a352ffffad3a05db4ff9d0c3aeacd5a33a: Status 404 returned error can't find the container with id 0fbc1ae183c4997441cc4b32259800a352ffffad3a05db4ff9d0c3aeacd5a33a
Nov 01 22:35:37.166231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:37.166204    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:35:37.166618 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:37.166508    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:35:37.167143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:37.167123    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 22:35:37.735490 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:37.735460    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" event={"ID":"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b","Type":"ContainerStarted","Data":"b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa"}
Nov 01 22:35:37.735651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:37.735496    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" event={"ID":"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b","Type":"ContainerStarted","Data":"0fbc1ae183c4997441cc4b32259800a352ffffad3a05db4ff9d0c3aeacd5a33a"}
Nov 01 22:35:37.844413 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:37.844368    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podStartSLOduration=1.8443549510000001 podStartE2EDuration="1.844354951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:37.842807693 +0000 UTC m=+719.432006753" watchObservedRunningTime="2026-11-01 22:35:37.844354951 +0000 UTC m=+719.433554012"
Nov 01 22:35:38.481467 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:38.481435    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:40.082407 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:40.082361    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:35:46.483153 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:46.483120    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:46.504816 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:46.504777    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:46.789860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:46.789779    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:35:47.212560 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.212529    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"]
Nov 01 22:35:47.225598 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.225566    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.273500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.273472    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"]
Nov 01 22:35:47.317443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.317420    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-a6dd2f-serving-cert\""
Nov 01 22:35:47.317611 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.317595    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 22:35:47.317655 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.317622    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 22:35:47.317693 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.317666    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xp822\""
Nov 01 22:35:47.322116 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.322099    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 22:35:47.405875 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.405846    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-metrics-certs\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.405875 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.405877    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ql9q\" (UniqueName: \"kubernetes.io/projected/fb963b29-0c1b-439c-a887-763eff140b30-kube-api-access-7ql9q\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.406128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.405912    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.406128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.406101    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb963b29-0c1b-439c-a887-763eff140b30-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.506940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.506867    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-metrics-certs\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.506940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.506906    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ql9q\" (UniqueName: \"kubernetes.io/projected/fb963b29-0c1b-439c-a887-763eff140b30-kube-api-access-7ql9q\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.507320 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.507081    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.507320 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.507138    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb963b29-0c1b-439c-a887-763eff140b30-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.507791 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.507765    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb963b29-0c1b-439c-a887-763eff140b30-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.509554 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.509535    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.509655 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.509579    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-metrics-certs\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.560316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.560293    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ql9q\" (UniqueName: \"kubernetes.io/projected/fb963b29-0c1b-439c-a887-763eff140b30-kube-api-access-7ql9q\") pod \"maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:47.839160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:47.839077    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:48.226844 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:48.226817    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"]
Nov 01 22:35:48.228242 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:35:48.228212    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb963b29_0c1b_439c_a887_763eff140b30.slice/crio-80367921bb488ca86884b11e4b84a8e6a2237f7bed7965c2baf538bf7600da21 WatchSource:0}: Error finding container 80367921bb488ca86884b11e4b84a8e6a2237f7bed7965c2baf538bf7600da21: Status 404 returned error can't find the container with id 80367921bb488ca86884b11e4b84a8e6a2237f7bed7965c2baf538bf7600da21
Nov 01 22:35:48.229677 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:48.229661    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:35:48.790338 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:48.790306    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc" event={"ID":"fb963b29-0c1b-439c-a887-763eff140b30","Type":"ContainerStarted","Data":"80367921bb488ca86884b11e4b84a8e6a2237f7bed7965c2baf538bf7600da21"}
Nov 01 22:35:48.816478 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:35:48.816447    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:35:50.804444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:50.804407    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc" event={"ID":"fb963b29-0c1b-439c-a887-763eff140b30","Type":"ContainerStarted","Data":"1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e"}
Nov 01 22:35:50.804841 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:50.804473    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:35:50.912805 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:50.912757    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc" podStartSLOduration=1.7901959779999999 podStartE2EDuration="3.912744631s" totalImagesPullingTime="2.122548653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:50.912428796 +0000 UTC m=+732.501627858" watchObservedRunningTime="2026-11-01 22:35:50.912744631 +0000 UTC m=+732.501943703"
Nov 01 22:35:51.645065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:51.628777    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:35:56.815209 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:35:56.815182    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 22:36:00.898078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:00.898036    4837 generic.go:356] "Generic (PLEG): container finished" podID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerID="aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d" exitCode=0
Nov 01 22:36:00.898430 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:00.898071    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" event={"ID":"85b5a76c-198e-4891-8383-a92528dcc6ca","Type":"ContainerDied","Data":"aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d"}
Nov 01 22:36:00.898494 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:00.898477    4837 scope.go:122] "RemoveContainer" containerID="aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d"
Nov 01 22:36:00.899985 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:00.899950    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:36:00.900138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:00.900059    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:36:00.900138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:00.900114    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:36:00.903156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:00.903137    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:36:01.904996 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:01.904965    4837 generic.go:356] "Generic (PLEG): container finished" podID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerID="f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a" exitCode=0
Nov 01 22:36:01.905488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:01.905036    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" event={"ID":"0562f9bc-b7b9-4885-9050-35e6200c19a7","Type":"ContainerDied","Data":"f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a"}
Nov 01 22:36:01.905488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:01.905482    4837 scope.go:122] "RemoveContainer" containerID="f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a"
Nov 01 22:36:01.907027 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:01.906995    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:36:01.907237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:01.907212    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" event={"ID":"85b5a76c-198e-4891-8383-a92528dcc6ca","Type":"ContainerStarted","Data":"f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0"}
Nov 01 22:36:01.907348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:01.907217    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:36:01.907421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:01.907368    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:36:02.541461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.541430    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:36:02.541818 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.541735    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 01 22:36:02.541986 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.541963    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 01 22:36:02.684160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.683998    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"]
Nov 01 22:36:02.688383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.688363    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.768108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.753098    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"premium-w1-5b8fd2-kserve-self-signed-certs\""
Nov 01 22:36:02.787122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.771614    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-home\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.787122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.771681    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9s2m\" (UniqueName: \"kubernetes.io/projected/e5890029-fa50-49f7-bf3f-0803dd176bcb-kube-api-access-j9s2m\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.787122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.771720    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-model-cache\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.787122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.771757    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-tmp-dir\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.787122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.771820    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-dshm\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.787122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.771844    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890029-fa50-49f7-bf3f-0803dd176bcb-tls-certs\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.787122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.773473    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"]
Nov 01 22:36:02.886119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.885724    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-dshm\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.886119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.885786    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890029-fa50-49f7-bf3f-0803dd176bcb-tls-certs\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.886119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.885903    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-home\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.886119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.885956    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9s2m\" (UniqueName: \"kubernetes.io/projected/e5890029-fa50-49f7-bf3f-0803dd176bcb-kube-api-access-j9s2m\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.886119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.885997    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-model-cache\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.886119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.886030    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-tmp-dir\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.902607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.901007    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-model-cache\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.902607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.901376    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-home\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.902607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.901622    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-tmp-dir\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.938066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.930645    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890029-fa50-49f7-bf3f-0803dd176bcb-tls-certs\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.960417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.960382    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-dshm\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:02.981967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.981132    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" event={"ID":"0562f9bc-b7b9-4885-9050-35e6200c19a7","Type":"ContainerStarted","Data":"a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f"}
Nov 01 22:36:02.981967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:02.981760    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9s2m\" (UniqueName: \"kubernetes.io/projected/e5890029-fa50-49f7-bf3f-0803dd176bcb-kube-api-access-j9s2m\") pod \"premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") " pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:03.000903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.000874    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:03.502624 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.502592    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"]
Nov 01 22:36:03.505017 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:36:03.504975    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5890029_fa50_49f7_bf3f_0803dd176bcb.slice/crio-6c4052ebc35cb5d5e3170197f135c2951d04e5cfab95c1a1a8c26aef2e46abf3 WatchSource:0}: Error finding container 6c4052ebc35cb5d5e3170197f135c2951d04e5cfab95c1a1a8c26aef2e46abf3: Status 404 returned error can't find the container with id 6c4052ebc35cb5d5e3170197f135c2951d04e5cfab95c1a1a8c26aef2e46abf3
Nov 01 22:36:03.517343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.517324    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:36:03.517616 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.517593    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 22:36:03.517702 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.517642    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 22:36:03.549439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.543381    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 01 22:36:03.549439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.543433    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 01 22:36:03.999798 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.999756    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" event={"ID":"e5890029-fa50-49f7-bf3f-0803dd176bcb","Type":"ContainerStarted","Data":"67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552"}
Nov 01 22:36:03.999798 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:03.999802    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" event={"ID":"e5890029-fa50-49f7-bf3f-0803dd176bcb","Type":"ContainerStarted","Data":"6c4052ebc35cb5d5e3170197f135c2951d04e5cfab95c1a1a8c26aef2e46abf3"}
Nov 01 22:36:04.002128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:04.002114    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:04.111768 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:04.111686    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podStartSLOduration=2.111668323 podStartE2EDuration="2.111668323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:04.110233601 +0000 UTC m=+745.699432661" watchObservedRunningTime="2026-11-01 22:36:04.111668323 +0000 UTC m=+745.700867386"
Nov 01 22:36:04.517480 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:04.517446    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 22:36:04.517691 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:04.517530    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 22:36:04.549066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:04.545369    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 01 22:36:04.549066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:04.545438    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 01 22:36:05.517457 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:05.517426    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 22:36:05.517828 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:05.517487    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 22:36:05.535078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:05.534330    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 01 22:36:05.535078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:05.534379    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 01 22:36:06.516958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:06.516922    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 22:36:06.517165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:06.517011    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 22:36:06.538809 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:06.534297    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 01 22:36:06.538809 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:06.534348    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 01 22:36:07.517313 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:07.517278    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 01 22:36:07.517489 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:07.517335    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 01 22:36:07.568556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:07.568523    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:36:07.570735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:07.570698    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:36:07.591199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:07.591165    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 22:36:08.525094 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:08.525059    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:36:08.525348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:08.525332    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:36:08.526013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:08.525993    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:36:13.001465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:13.001424    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:13.014178 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:13.014149    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:13.069001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:13.068974    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:36:17.447922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:17.447887    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:36:18.814385 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:36:18.814351    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:36:26.025886 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.025841    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"]
Nov 01 22:36:26.029822 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.029802    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.125164 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.125132    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"kube-root-ca.crt\""
Nov 01 22:36:26.125340 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.125186    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"default-dockercfg-nh9pl\""
Nov 01 22:36:26.125340 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.125316    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"openshift-service-ca.crt\""
Nov 01 22:36:26.125475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.125354    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"simulator-w3-e71d55-kserve-self-signed-certs\""
Nov 01 22:36:26.127600 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.127573    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"]
Nov 01 22:36:26.169649 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.169547    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-home\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.171911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.171190    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f066a460-1722-4886-9978-8f677610c83b-tls-certs\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.171911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.171264    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-model-cache\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.171911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.171346    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-tmp-dir\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.171911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.171371    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96w9t\" (UniqueName: \"kubernetes.io/projected/f066a460-1722-4886-9978-8f677610c83b-kube-api-access-96w9t\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.171911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.171418    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-dshm\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.272986 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.272951    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f066a460-1722-4886-9978-8f677610c83b-tls-certs\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.273199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.273001    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-model-cache\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.273199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.273078    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-tmp-dir\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.273199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.273100    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96w9t\" (UniqueName: \"kubernetes.io/projected/f066a460-1722-4886-9978-8f677610c83b-kube-api-access-96w9t\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.273199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.273126    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-dshm\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.273499 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.273238    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-home\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.273499 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.273404    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-model-cache\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.273499 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.273482    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-tmp-dir\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.273615 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.273554    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-home\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.275305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.275282    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-dshm\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.275633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.275616    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f066a460-1722-4886-9978-8f677610c83b-tls-certs\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.326927 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.326863    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96w9t\" (UniqueName: \"kubernetes.io/projected/f066a460-1722-4886-9978-8f677610c83b-kube-api-access-96w9t\") pod \"simulator-w3-e71d55-kserve-54d4c876bc-4qcmd\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") " pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.341683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.341652    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:26.712344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:26.712322    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"]
Nov 01 22:36:26.714169 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:36:26.714143    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf066a460_1722_4886_9978_8f677610c83b.slice/crio-4108d4fd7fb66ecc19534d05f126648b1a36183d230d880ae159774170128bd0 WatchSource:0}: Error finding container 4108d4fd7fb66ecc19534d05f126648b1a36183d230d880ae159774170128bd0: Status 404 returned error can't find the container with id 4108d4fd7fb66ecc19534d05f126648b1a36183d230d880ae159774170128bd0
Nov 01 22:36:27.138657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:27.138581    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" event={"ID":"f066a460-1722-4886-9978-8f677610c83b","Type":"ContainerStarted","Data":"f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd"}
Nov 01 22:36:27.138657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:27.138619    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" event={"ID":"f066a460-1722-4886-9978-8f677610c83b","Type":"ContainerStarted","Data":"4108d4fd7fb66ecc19534d05f126648b1a36183d230d880ae159774170128bd0"}
Nov 01 22:36:27.256017 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:27.255961    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podStartSLOduration=2.255946252 podStartE2EDuration="2.255946252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:27.25351872 +0000 UTC m=+768.842717810" watchObservedRunningTime="2026-11-01 22:36:27.255946252 +0000 UTC m=+768.845145307"
Nov 01 22:36:27.342330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:27.342297    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:31.116028 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:31.115994    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:36:36.342859 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:36.342827    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:36.355506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:36.355485    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:37.214157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:37.214127    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:36:48.814852 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:36:48.814815    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:36:59.944250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:36:59.944214    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:37:08.969061 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:08.969009    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"]
Nov 01 22:37:08.973136 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:08.973116    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.030510 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.030482    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"openshift-service-ca.crt\""
Nov 01 22:37:09.068952 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.068928    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"default-dockercfg-qgkfn\""
Nov 01 22:37:09.069110 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.069028    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"kube-root-ca.crt\""
Nov 01 22:37:09.069172 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.069113    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"simulator-w0-c54de0-kserve-self-signed-certs\""
Nov 01 22:37:09.070405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.070382    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-model-cache\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.070514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.070449    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tmp-dir\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.070562 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.070513    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgz6\" (UniqueName: \"kubernetes.io/projected/cb48bd1c-2d40-4e8e-93ae-a6735406208d-kube-api-access-5jgz6\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.070609 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.070583    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-dshm\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.070665 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.070654    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tls-certs\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.070702 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.070695    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-home\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.071175 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.071155    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"]
Nov 01 22:37:09.171507 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.171472    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tls-certs\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.171679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.171545    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-home\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.171679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.171588    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-model-cache\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.171679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.171655    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tmp-dir\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.171846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.171682    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgz6\" (UniqueName: \"kubernetes.io/projected/cb48bd1c-2d40-4e8e-93ae-a6735406208d-kube-api-access-5jgz6\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.171846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.171775    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-dshm\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.171958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.171931    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-home\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.172025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.171991    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-model-cache\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.172105 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.172064    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tmp-dir\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.173924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.173898    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-dshm\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.174253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.174234    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tls-certs\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.225986 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.225929    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgz6\" (UniqueName: \"kubernetes.io/projected/cb48bd1c-2d40-4e8e-93ae-a6735406208d-kube-api-access-5jgz6\") pod \"simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") " pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.285285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.285258    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:09.651507 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:09.651481    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"]
Nov 01 22:37:09.652447 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:37:09.652419    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb48bd1c_2d40_4e8e_93ae_a6735406208d.slice/crio-797c10e2c0940dc52928ec5aa5201f17687a1becbcfeee9edd5816683c43ec53 WatchSource:0}: Error finding container 797c10e2c0940dc52928ec5aa5201f17687a1becbcfeee9edd5816683c43ec53: Status 404 returned error can't find the container with id 797c10e2c0940dc52928ec5aa5201f17687a1becbcfeee9edd5816683c43ec53
Nov 01 22:37:10.371563 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:10.371522    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" event={"ID":"cb48bd1c-2d40-4e8e-93ae-a6735406208d","Type":"ContainerStarted","Data":"54c0579e972a27c1a16250a3aa725248edee580ee03645915b0790266c7dd4af"}
Nov 01 22:37:10.371563 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:10.371567    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" event={"ID":"cb48bd1c-2d40-4e8e-93ae-a6735406208d","Type":"ContainerStarted","Data":"797c10e2c0940dc52928ec5aa5201f17687a1becbcfeee9edd5816683c43ec53"}
Nov 01 22:37:10.485134 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:10.485084    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podStartSLOduration=2.485069759 podStartE2EDuration="2.485069759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:10.483795269 +0000 UTC m=+812.072994334" watchObservedRunningTime="2026-11-01 22:37:10.485069759 +0000 UTC m=+812.074268819"
Nov 01 22:37:11.285363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:11.285330    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:18.433526 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:18.433492    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:37:18.814805 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:37:18.814718    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:37:19.285800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:19.285769    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:19.297655 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:19.297633    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:19.430451 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:19.430428    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:37:24.268666 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.268628    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"]
Nov 01 22:37:24.272623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.272599    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321571 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321544    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321587    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321614    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321718    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321760    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321778    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321961 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321854    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321961 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321902    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48v9q\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-kube-api-access-48v9q\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.321961 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.321935    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.368157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.368133    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5b8fd2-openshift-default-dockercfg-bqfks\""
Nov 01 22:37:24.371569 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.371501    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"]
Nov 01 22:37:24.422771 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.422744    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48v9q\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-kube-api-access-48v9q\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.422903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.422785    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.422903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.422845    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.422901    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.422934    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.422983    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.423017    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.423057    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.423117    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423378 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.423324    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423585 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.423562    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.423651    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.423658    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.423876 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.423854    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.425660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.425634    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.425817 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.425797    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.476865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.476843    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.476865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.476852    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48v9q\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-kube-api-access-48v9q\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:24.585820 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:24.585772    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:25.000272 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:25.000244    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"]
Nov 01 22:37:25.035683 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:37:25.035643    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5324fb8a_e6de_48cb_88e7_db3d2280ff12.slice/crio-1835f45ddf5d4a2402c857f561d8745d05fdefc2e539210b23442a74465a49f7 WatchSource:0}: Error finding container 1835f45ddf5d4a2402c857f561d8745d05fdefc2e539210b23442a74465a49f7: Status 404 returned error can't find the container with id 1835f45ddf5d4a2402c857f561d8745d05fdefc2e539210b23442a74465a49f7
Nov 01 22:37:25.038363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:25.038324    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:37:25.038456 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:25.038410    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:37:25.038494 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:25.038451    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:37:25.464826 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:25.464794    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" event={"ID":"5324fb8a-e6de-48cb-88e7-db3d2280ff12","Type":"ContainerStarted","Data":"eae58a75f219598fe97aa7a7b1367c424992baa8148da0bd08bd2da9e4d154cb"}
Nov 01 22:37:25.465253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:25.464843    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" event={"ID":"5324fb8a-e6de-48cb-88e7-db3d2280ff12","Type":"ContainerStarted","Data":"1835f45ddf5d4a2402c857f561d8745d05fdefc2e539210b23442a74465a49f7"}
Nov 01 22:37:25.578624 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:25.578574    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" podStartSLOduration=1.5785590470000002 podStartE2EDuration="1.578559047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:25.576280278 +0000 UTC m=+827.165479339" watchObservedRunningTime="2026-11-01 22:37:25.578559047 +0000 UTC m=+827.167758107"
Nov 01 22:37:25.586298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:25.586273    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:26.588109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:26.588077    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 01 22:37:26.588506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:26.588136    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 01 22:37:27.586757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:27.586727    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 01 22:37:27.586925 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:27.586778    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 01 22:37:28.586754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:28.586716    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 01 22:37:28.587145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:28.586790    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 01 22:37:29.586207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:29.586168    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 01 22:37:29.586524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:29.586240    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 01 22:37:30.586143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:30.586104    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 01 22:37:30.586561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:30.586170    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 01 22:37:31.697416 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:31.697384    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:31.698175 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:31.698152    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:31.699164 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:31.699140    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 22:37:47.849164 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:37:47.849089    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:37:48.811116 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:37:48.811032    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:38:03.848314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.848285    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"]
Nov 01 22:38:03.852279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.852260    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:03.912536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.912498    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"openshift-service-ca.crt\""
Nov 01 22:38:03.947936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.947912    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"premium-w2-c768f3-kserve-self-signed-certs\""
Nov 01 22:38:03.948181 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.948160    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"default-dockercfg-l6sl9\""
Nov 01 22:38:03.950441 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.950421    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"]
Nov 01 22:38:03.955638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.955618    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-model-cache\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:03.955715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.955699    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-tmp-dir\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:03.955753 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.955742    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-dshm\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:03.955870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.955855    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-home\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:03.955909 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.955889    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/53d01282-ad46-4cc0-bf80-3113fad5392e-kube-api-access-824ts\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:03.955943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:03.955912    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d01282-ad46-4cc0-bf80-3113fad5392e-tls-certs\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.002583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.002537    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"kube-root-ca.crt\""
Nov 01 22:38:04.057453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.057421    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/53d01282-ad46-4cc0-bf80-3113fad5392e-kube-api-access-824ts\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.057633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.057469    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d01282-ad46-4cc0-bf80-3113fad5392e-tls-certs\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.057633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.057512    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-model-cache\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.057633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.057574    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-tmp-dir\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.057633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.057607    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-dshm\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.057835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.057656    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-home\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.058021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.057991    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-model-cache\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.058177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.058102    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-tmp-dir\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.058177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.058141    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-home\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.060102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.060079    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d01282-ad46-4cc0-bf80-3113fad5392e-tls-certs\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.060194 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.060161    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-dshm\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.111663 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.111599    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/53d01282-ad46-4cc0-bf80-3113fad5392e-kube-api-access-824ts\") pod \"premium-w2-c768f3-kserve-747bf497c7-hnmf7\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") " pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.163540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.163518    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:04.576531 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.576509    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"]
Nov 01 22:38:04.578191 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:38:04.578167    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53d01282_ad46_4cc0_bf80_3113fad5392e.slice/crio-25ce110e10f10de36d30d38248c03e8dcfc8877bac53a6aa7c6a0539c86a6974 WatchSource:0}: Error finding container 25ce110e10f10de36d30d38248c03e8dcfc8877bac53a6aa7c6a0539c86a6974: Status 404 returned error can't find the container with id 25ce110e10f10de36d30d38248c03e8dcfc8877bac53a6aa7c6a0539c86a6974
Nov 01 22:38:04.683636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.683604    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" event={"ID":"53d01282-ad46-4cc0-bf80-3113fad5392e","Type":"ContainerStarted","Data":"95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b"}
Nov 01 22:38:04.683767 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.683651    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" event={"ID":"53d01282-ad46-4cc0-bf80-3113fad5392e","Type":"ContainerStarted","Data":"25ce110e10f10de36d30d38248c03e8dcfc8877bac53a6aa7c6a0539c86a6974"}
Nov 01 22:38:04.794450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:04.794398    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podStartSLOduration=1.79438467 podStartE2EDuration="1.79438467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:04.791645055 +0000 UTC m=+866.380844115" watchObservedRunningTime="2026-11-01 22:38:04.79438467 +0000 UTC m=+866.383583730"
Nov 01 22:38:05.163905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:05.163880    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:13.200242 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:13.200207    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:38:14.164084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:14.164050    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:14.176580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:14.176551    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:14.748514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:14.748482    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:38:18.811138 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:38:18.811110    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:38:38.611607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:38.611581    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:38:38.614401 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:38.614380    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:38:43.775200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:43.775129    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:38:47.758632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.758601    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"]
Nov 01 22:38:47.764174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.764156    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.787510 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.787487    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-home\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.787645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.787531    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjghj\" (UniqueName: \"kubernetes.io/projected/0fbe9044-faad-4c63-bc0d-292c2e910a2d-kube-api-access-hjghj\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.787645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.787568    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-dshm\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.787749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.787718    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tmp-dir\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.787800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.787779    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tls-certs\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.787911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.787895    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-model-cache\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.816989 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.816961    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"kube-root-ca.crt\""
Nov 01 22:38:47.816989 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.816980    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"openshift-service-ca.crt\""
Nov 01 22:38:47.863476 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.863447    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"premium-w6-a6dd2f-kserve-self-signed-certs\""
Nov 01 22:38:47.863622 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.863508    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"default-dockercfg-gptqw\""
Nov 01 22:38:47.865950 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.865933    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"]
Nov 01 22:38:47.888657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.888624    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-home\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.888657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.888673    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjghj\" (UniqueName: \"kubernetes.io/projected/0fbe9044-faad-4c63-bc0d-292c2e910a2d-kube-api-access-hjghj\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.888835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.888696    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-dshm\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.888835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.888752    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tmp-dir\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.888835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.888793    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tls-certs\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.889012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.888868    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-model-cache\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.889099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.889067    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-home\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.889200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.889181    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tmp-dir\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.889253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.889211    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-model-cache\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.890997 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.890981    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-dshm\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.891228 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.891210    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tls-certs\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:47.950810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:47.949410    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjghj\" (UniqueName: \"kubernetes.io/projected/0fbe9044-faad-4c63-bc0d-292c2e910a2d-kube-api-access-hjghj\") pod \"premium-w6-a6dd2f-kserve-646d496bd9-8l5w9\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") " pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:48.075443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:48.075353    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:48.486599 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:48.486574    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"]
Nov 01 22:38:48.487699 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:38:48.487666    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fbe9044_faad_4c63_bc0d_292c2e910a2d.slice/crio-4d2123779aa0536fb5d9016609dd32fbb5e67a88f9e3276b653ed95ebc2d346d WatchSource:0}: Error finding container 4d2123779aa0536fb5d9016609dd32fbb5e67a88f9e3276b653ed95ebc2d346d: Status 404 returned error can't find the container with id 4d2123779aa0536fb5d9016609dd32fbb5e67a88f9e3276b653ed95ebc2d346d
Nov 01 22:38:48.812526 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:38:48.812447    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:38:48.963259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:48.963215    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" event={"ID":"0fbe9044-faad-4c63-bc0d-292c2e910a2d","Type":"ContainerStarted","Data":"e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495"}
Nov 01 22:38:48.963417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:48.963263    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" event={"ID":"0fbe9044-faad-4c63-bc0d-292c2e910a2d","Type":"ContainerStarted","Data":"4d2123779aa0536fb5d9016609dd32fbb5e67a88f9e3276b653ed95ebc2d346d"}
Nov 01 22:38:49.070976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:49.070882    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podStartSLOduration=2.070869889 podStartE2EDuration="2.070869889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:49.070517393 +0000 UTC m=+910.659716472" watchObservedRunningTime="2026-11-01 22:38:49.070869889 +0000 UTC m=+910.660068949"
Nov 01 22:38:49.075929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:49.075905    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:58.075905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:58.075872    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:58.088285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:58.088263    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:38:59.025915 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:38:59.025887    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:39:10.003204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.003174    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"]
Nov 01 22:39:10.007270 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.007248    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.142667 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.142631    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.142979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.142950    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.143165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.142997    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.143165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.143100    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.143165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.143135    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.143336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.143168    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs9k2\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-kube-api-access-xs9k2\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.143336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.143200    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.143336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.143254    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.143336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.143274    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e14863a-1388-4ccd-8c62-8630cc3ba882-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.155952 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.155928    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"]
Nov 01 22:39:10.256824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.256656    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.257258 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.257209    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.257412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.257288    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9k2\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-kube-api-access-xs9k2\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.257412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.257361    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.257539 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.257474    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.257539 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.257509    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e14863a-1388-4ccd-8c62-8630cc3ba882-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.257687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.257659    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.257924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.257903    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.258113 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.258094    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.258355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.258330    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.258896 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.258862    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.258994 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.258905    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e14863a-1388-4ccd-8c62-8630cc3ba882-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.259317 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.259292    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.259434 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.259415    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.280071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.276475    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.290778 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.290731    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.311936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.311907    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.312129 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.312035    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9k2\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-kube-api-access-xs9k2\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.321869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.321826    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:10.734624 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.734599    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"]
Nov 01 22:39:10.736215 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:39:10.736177    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e14863a_1388_4ccd_8c62_8630cc3ba882.slice/crio-4873829735a0556bacbf68cb668a43a7f565ba1ede91dacf1b820e6988c0b249 WatchSource:0}: Error finding container 4873829735a0556bacbf68cb668a43a7f565ba1ede91dacf1b820e6988c0b249: Status 404 returned error can't find the container with id 4873829735a0556bacbf68cb668a43a7f565ba1ede91dacf1b820e6988c0b249
Nov 01 22:39:10.738534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.738496    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:10.738650 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.738563    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:10.738650 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:10.738600    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:11.086089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:11.085988    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" event={"ID":"8e14863a-1388-4ccd-8c62-8630cc3ba882","Type":"ContainerStarted","Data":"cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc"}
Nov 01 22:39:11.086089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:11.086058    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" event={"ID":"8e14863a-1388-4ccd-8c62-8630cc3ba882","Type":"ContainerStarted","Data":"4873829735a0556bacbf68cb668a43a7f565ba1ede91dacf1b820e6988c0b249"}
Nov 01 22:39:11.199215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:11.199142    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" podStartSLOduration=2.1991214279999998 podStartE2EDuration="2.199121428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:11.197338424 +0000 UTC m=+932.786537486" watchObservedRunningTime="2026-11-01 22:39:11.199121428 +0000 UTC m=+932.788320492"
Nov 01 22:39:11.322695 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:11.322664    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:11.324628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:11.324601    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:39:11.324756 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:11.324648    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 22:39:12.322903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:12.322862    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:39:12.323303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:12.322920    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 22:39:13.322293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:13.322256    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:39:13.322478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:13.322321    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 22:39:14.322397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:14.322364    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:39:14.322792 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:14.322419    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 22:39:15.322678 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:15.322644    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:39:15.323166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:15.322696    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 22:39:16.322844 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:16.322805    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:39:16.323269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:16.322881    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 22:39:16.930853 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:16.930810    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:39:17.421948 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:17.421920    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:17.422474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:17.422390    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:17.423539 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:17.423508    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 22:39:18.812690 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:39:18.812659    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:39:24.887553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.887520    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"]
Nov 01 22:39:24.891453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.891433    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:24.959573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.959538    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"]
Nov 01 22:39:24.960357 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.960324    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy" containerID="cri-o://3cdc2b41f29e9c5254be08aeb0c8c607319c4ae6eec484da6eec3647e9af3bb5" gracePeriod=30
Nov 01 22:39:24.965881 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.965857    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"]
Nov 01 22:39:24.966439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.966412    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy" containerID="cri-o://4868427d35c196e4cc590e1ca72fbdaa0772ab578f815b9368581c11191aa431" gracePeriod=30
Nov 01 22:39:24.987108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.984514    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a6dd2f-openshift-default-dockercfg-vvpcs\""
Nov 01 22:39:24.989125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.989028    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"]
Nov 01 22:39:24.989518 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.989487    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" containerID="cri-o://12faf3d05567e3cdd100455242986f36ae3b04dd85ba72a306d92ebb66b3eae0" gracePeriod=30
Nov 01 22:39:24.995948 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:24.995922    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"]
Nov 01 22:39:25.027211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027178    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.027343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027214    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.027343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027245    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.027343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027314    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.027514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027363    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.027514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027378    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.027514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027398    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.027514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027427    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.027514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.027450    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2nx9\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-kube-api-access-d2nx9\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.064560 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.064524    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-6z8hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:39:25.064678 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.064579    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:39:25.128375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128339    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128378    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128598 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128496    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128598 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128543    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128598 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128569    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2nx9\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-kube-api-access-d2nx9\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128619    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128643    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128690    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128933 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128801    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.128933 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.128897    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.129091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.129017    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.129206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.129183    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.129287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.129257    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.129413 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.129396    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.130816 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.130775    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.131131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.131111    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.182410 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.182378    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.182652 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.182631    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2nx9\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-kube-api-access-d2nx9\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.205766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.205747    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:25.281243 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.279920    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"]
Nov 01 22:39:25.291759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.291737    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.333307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.333279    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"]
Nov 01 22:39:25.340474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.340446    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:39:25.340594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.340521    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:39:25.432558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.432479    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.432682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.432629    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.432682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.432665    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcsr\" (UniqueName: \"kubernetes.io/projected/66f567f2-244a-4f74-9b23-4b4949b6ac10-kube-api-access-qfcsr\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.533585 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.533558    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.533753 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.533599    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcsr\" (UniqueName: \"kubernetes.io/projected/66f567f2-244a-4f74-9b23-4b4949b6ac10-kube-api-access-qfcsr\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.533753 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.533648    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.534328 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.534306    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.535927 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.535908    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.588669 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.588638    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcsr\" (UniqueName: \"kubernetes.io/projected/66f567f2-244a-4f74-9b23-4b4949b6ac10-kube-api-access-qfcsr\") pod \"maas-api-revoke-sub-e04d8cc2-sczqt\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.607365 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.607337    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:25.618411 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.618390    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"]
Nov 01 22:39:25.619709 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:39:25.619567    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7eee6d_c6ff_440e_bade_d62562a6c57a.slice/crio-704147f5b24c703ca40245488a099a5385a38263839a234731a95162259ddfe7 WatchSource:0}: Error finding container 704147f5b24c703ca40245488a099a5385a38263839a234731a95162259ddfe7: Status 404 returned error can't find the container with id 704147f5b24c703ca40245488a099a5385a38263839a234731a95162259ddfe7
Nov 01 22:39:25.622168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.622138    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:25.622237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.622205    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:25.622280 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.622236    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:25.975693 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:25.975662    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"]
Nov 01 22:39:25.976425 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:39:25.976387    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f567f2_244a_4f74_9b23_4b4949b6ac10.slice/crio-fac5b2f428fb505e52573ec1388bd32ba963a3aa844b6b48e53e6df646b3002e WatchSource:0}: Error finding container fac5b2f428fb505e52573ec1388bd32ba963a3aa844b6b48e53e6df646b3002e: Status 404 returned error can't find the container with id fac5b2f428fb505e52573ec1388bd32ba963a3aa844b6b48e53e6df646b3002e
Nov 01 22:39:26.186782 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:26.186748    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt" event={"ID":"66f567f2-244a-4f74-9b23-4b4949b6ac10","Type":"ContainerStarted","Data":"fac5b2f428fb505e52573ec1388bd32ba963a3aa844b6b48e53e6df646b3002e"}
Nov 01 22:39:26.188370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:26.188341    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" event={"ID":"5d7eee6d-c6ff-440e-bade-d62562a6c57a","Type":"ContainerStarted","Data":"9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57"}
Nov 01 22:39:26.188492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:26.188395    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" event={"ID":"5d7eee6d-c6ff-440e-bade-d62562a6c57a","Type":"ContainerStarted","Data":"704147f5b24c703ca40245488a099a5385a38263839a234731a95162259ddfe7"}
Nov 01 22:39:26.206728 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:26.206709    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:26.208000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:26.207968    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:39:26.208098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:26.208026    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 22:39:26.305443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:26.305345    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podStartSLOduration=2.305330744 podStartE2EDuration="2.305330744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:26.302551027 +0000 UTC m=+947.891750087" watchObservedRunningTime="2026-11-01 22:39:26.305330744 +0000 UTC m=+947.894529805"
Nov 01 22:39:27.206926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:27.206894    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:39:27.207390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:27.206954    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 22:39:28.206908 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:28.206874    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:39:28.207086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:28.206941    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 22:39:29.206100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:29.206070    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:39:29.206271 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:29.206121    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 22:39:30.208071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.206926    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:39:30.208071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.207003    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 22:39:30.218088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.217010    4837 generic.go:356] "Generic (PLEG): container finished" podID="535a603e-c963-4fc8-a816-2e1b8209120e" containerID="3cdc2b41f29e9c5254be08aeb0c8c607319c4ae6eec484da6eec3647e9af3bb5" exitCode=0
Nov 01 22:39:30.218088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.217154    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" event={"ID":"535a603e-c963-4fc8-a816-2e1b8209120e","Type":"ContainerDied","Data":"3cdc2b41f29e9c5254be08aeb0c8c607319c4ae6eec484da6eec3647e9af3bb5"}
Nov 01 22:39:30.219343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.219318    4837 generic.go:356] "Generic (PLEG): container finished" podID="832a757e-f762-446b-aa18-5425bad1c560" containerID="12faf3d05567e3cdd100455242986f36ae3b04dd85ba72a306d92ebb66b3eae0" exitCode=0
Nov 01 22:39:30.219471 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.219458    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" event={"ID":"832a757e-f762-446b-aa18-5425bad1c560","Type":"ContainerDied","Data":"12faf3d05567e3cdd100455242986f36ae3b04dd85ba72a306d92ebb66b3eae0"}
Nov 01 22:39:30.222186 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.222162    4837 generic.go:356] "Generic (PLEG): container finished" podID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerID="4868427d35c196e4cc590e1ca72fbdaa0772ab578f815b9368581c11191aa431" exitCode=0
Nov 01 22:39:30.222275 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.222220    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" event={"ID":"551392b2-5288-4eda-8e42-3bf42fb8b7d2","Type":"ContainerDied","Data":"4868427d35c196e4cc590e1ca72fbdaa0772ab578f815b9368581c11191aa431"}
Nov 01 22:39:30.313313 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.313284    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:39:30.402220 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.402190    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:39:30.406203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.406181    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:39:30.408031 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408012    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-kube-api-access-7fx42\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-kube-api-access-7fx42\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408142 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408095    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/535a603e-c963-4fc8-a816-2e1b8209120e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535a603e-c963-4fc8-a816-2e1b8209120e-istio-podinfo\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408142 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408136    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-credential-socket\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408310 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408164    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535a603e-c963-4fc8-a816-2e1b8209120e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535a603e-c963-4fc8-a816-2e1b8209120e-istiod-ca-cert\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408310 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408234    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-envoy\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408368    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-istio-token\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-istio-token\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408412    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-data\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408529 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408437    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-credential-socket" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.408529 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408457    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-socket\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408549    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-certs\") pod \"535a603e-c963-4fc8-a816-2e1b8209120e\" (UID: \"535a603e-c963-4fc8-a816-2e1b8209120e\") "
Nov 01 22:39:30.408634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408576    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535a603e-c963-4fc8-a816-2e1b8209120e-istiod-ca-cert" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:30.408848 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408824    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-socket" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.408930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.408831    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-data" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.409039 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.409018    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-certs" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.409128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.409033    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.409128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.409091    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535a603e-c963-4fc8-a816-2e1b8209120e-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.409128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.409107    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.409128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.409120    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.410876 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.410844    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-envoy" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.410981 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.410906    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-istio-token" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:30.410981 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.410907    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/535a603e-c963-4fc8-a816-2e1b8209120e-istio-podinfo" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:39:30.410981 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.410949    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-kube-api-access-7fx42" pod "535a603e-c963-4fc8-a816-2e1b8209120e" (UID: "535a603e-c963-4fc8-a816-2e1b8209120e"). InnerVolumeSpecName "kube-api-access-7fx42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:30.510055 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510012    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-istio-token\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-istio-token\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.510230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510069    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-token\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-token\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.510230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510104    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istiod-ca-cert\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.510230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510146    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-socket\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.510418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510390    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-socket" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.510532 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510511    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-kube-api-access-sl72l\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-kube-api-access-sl72l\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.510609 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510553    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-certs\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.510609 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510587    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-envoy\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.510723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510663    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-podinfo\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.510723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510713    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832a757e-f762-446b-aa18-5425bad1c560-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a757e-f762-446b-aa18-5425bad1c560-istiod-ca-cert\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.510821 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510745    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-data\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.510821 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510776    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-socket\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.510821 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510815    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/832a757e-f762-446b-aa18-5425bad1c560-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a757e-f762-446b-aa18-5425bad1c560-istio-podinfo\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.510971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510847    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-credential-socket\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.510971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510913    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-envoy\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.510971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510924    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istiod-ca-cert" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:30.511149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.510992    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-kube-api-access-4v92h\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-kube-api-access-4v92h\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.511149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.511036    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-credential-socket\") pod \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\" (UID: \"551392b2-5288-4eda-8e42-3bf42fb8b7d2\") "
Nov 01 22:39:30.511149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.511114    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-certs\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.511297 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.511162    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-data\") pod \"832a757e-f762-446b-aa18-5425bad1c560\" (UID: \"832a757e-f762-446b-aa18-5425bad1c560\") "
Nov 01 22:39:30.511504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.511383    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832a757e-f762-446b-aa18-5425bad1c560-istiod-ca-cert" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:30.511788 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.511594    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-data" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.511788 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.511620    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-credential-socket" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.511922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.511831    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-certs" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512034    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512091    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512106    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512120    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512136    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512151    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a757e-f762-446b-aa18-5425bad1c560-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512165    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535a603e-c963-4fc8-a816-2e1b8209120e-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512180    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512195    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fx42\" (UniqueName: \"kubernetes.io/projected/535a603e-c963-4fc8-a816-2e1b8209120e-kube-api-access-7fx42\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512211    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535a603e-c963-4fc8-a816-2e1b8209120e-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.512972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512857    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-socket" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.512972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.512929    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-istio-token" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:30.513229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.513198    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-certs" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.513292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.513221    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-credential-socket" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.513292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.513251    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-data" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.516635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.516566    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-podinfo" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:39:30.516635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.516578    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-kube-api-access-4v92h" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "kube-api-access-4v92h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:30.516770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.516688    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-token" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:30.518805 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.518454    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/832a757e-f762-446b-aa18-5425bad1c560-istio-podinfo" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:39:30.524824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.524755    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-envoy" pod "551392b2-5288-4eda-8e42-3bf42fb8b7d2" (UID: "551392b2-5288-4eda-8e42-3bf42fb8b7d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.524924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.524815    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-kube-api-access-sl72l" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "kube-api-access-sl72l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:30.526891 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.526684    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-envoy" pod "832a757e-f762-446b-aa18-5425bad1c560" (UID: "832a757e-f762-446b-aa18-5425bad1c560"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:39:30.613779 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.613742    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v92h\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-kube-api-access-4v92h\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.613981 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.613958    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614109    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614221    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614323 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614310    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614407 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614394    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl72l\" (UniqueName: \"kubernetes.io/projected/832a757e-f762-446b-aa18-5425bad1c560-kube-api-access-sl72l\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614496 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614484    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614581 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614569    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a757e-f762-446b-aa18-5425bad1c560-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614661    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614679    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614693    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614706    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a757e-f762-446b-aa18-5425bad1c560-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.614741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:30.614720    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551392b2-5288-4eda-8e42-3bf42fb8b7d2-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:31.206965 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.206937    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:39:31.207109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.207002    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 22:39:31.246079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.230267    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"
Nov 01 22:39:31.246079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.230517    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz" event={"ID":"535a603e-c963-4fc8-a816-2e1b8209120e","Type":"ContainerDied","Data":"2f1cbd141808feca7e7b8edebe546cbbbeac4178a0f809f1301d65d950ee7f80"}
Nov 01 22:39:31.246079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.230578    4837 scope.go:122] "RemoveContainer" containerID="3cdc2b41f29e9c5254be08aeb0c8c607319c4ae6eec484da6eec3647e9af3bb5"
Nov 01 22:39:31.258344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.258314    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk" event={"ID":"832a757e-f762-446b-aa18-5425bad1c560","Type":"ContainerDied","Data":"42f8ed8cd4fbb726d17c88abc964b890ca8545c262cbbb5e5d02aa344eb33635"}
Nov 01 22:39:31.258439 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.258431    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"
Nov 01 22:39:31.260956 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.260917    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf" event={"ID":"551392b2-5288-4eda-8e42-3bf42fb8b7d2","Type":"ContainerDied","Data":"7e505ae732a6674e9194af4135657490bcfcd67a53e7ada2c02e87902da2a4dc"}
Nov 01 22:39:31.261078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.261023    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"
Nov 01 22:39:31.266493 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.266474    4837 scope.go:122] "RemoveContainer" containerID="12faf3d05567e3cdd100455242986f36ae3b04dd85ba72a306d92ebb66b3eae0"
Nov 01 22:39:31.274760 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.274743    4837 scope.go:122] "RemoveContainer" containerID="4868427d35c196e4cc590e1ca72fbdaa0772ab578f815b9368581c11191aa431"
Nov 01 22:39:31.385809 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.385783    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"]
Nov 01 22:39:31.389935 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.389914    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-6z8hk"]
Nov 01 22:39:31.495414 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.495389    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"]
Nov 01 22:39:31.498912 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.498890    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pnpf"]
Nov 01 22:39:31.602361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.602265    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"]
Nov 01 22:39:31.609653 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:31.609620    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h49jz"]
Nov 01 22:39:32.273081 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:32.273029    4837 generic.go:356] "Generic (PLEG): container finished" podID="66f567f2-244a-4f74-9b23-4b4949b6ac10" containerID="8970f2e226792dff33cba56f5f44577211b09343f8da6af053d3cb07d879fe02" exitCode=0
Nov 01 22:39:32.273458 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:32.273095    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt" event={"ID":"66f567f2-244a-4f74-9b23-4b4949b6ac10","Type":"ContainerDied","Data":"8970f2e226792dff33cba56f5f44577211b09343f8da6af053d3cb07d879fe02"}
Nov 01 22:39:32.353078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:32.352981    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:32.353450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:32.353422    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:32.355618 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:32.355599    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:39:32.612665 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:32.612583    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" path="/var/lib/kubelet/pods/535a603e-c963-4fc8-a816-2e1b8209120e/volumes"
Nov 01 22:39:32.613590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:32.613568    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" path="/var/lib/kubelet/pods/551392b2-5288-4eda-8e42-3bf42fb8b7d2/volumes"
Nov 01 22:39:32.614371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:32.614353    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832a757e-f762-446b-aa18-5425bad1c560" path="/var/lib/kubelet/pods/832a757e-f762-446b-aa18-5425bad1c560/volumes"
Nov 01 22:39:33.554109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.551195    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:33.714078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.713011    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f567f2-244a-4f74-9b23-4b4949b6ac10-kube-api-access-qfcsr\" (UniqueName: \"kubernetes.io/projected/66f567f2-244a-4f74-9b23-4b4949b6ac10-kube-api-access-qfcsr\") pod \"66f567f2-244a-4f74-9b23-4b4949b6ac10\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") "
Nov 01 22:39:33.714078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.713088    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-xks-ca\") pod \"66f567f2-244a-4f74-9b23-4b4949b6ac10\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") "
Nov 01 22:39:33.714078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.713123    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-openshift-ca\") pod \"66f567f2-244a-4f74-9b23-4b4949b6ac10\" (UID: \"66f567f2-244a-4f74-9b23-4b4949b6ac10\") "
Nov 01 22:39:33.714428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.714406    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-openshift-ca" pod "66f567f2-244a-4f74-9b23-4b4949b6ac10" (UID: "66f567f2-244a-4f74-9b23-4b4949b6ac10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:33.715810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.715784    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-xks-ca" pod "66f567f2-244a-4f74-9b23-4b4949b6ac10" (UID: "66f567f2-244a-4f74-9b23-4b4949b6ac10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:33.715908 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.715810    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f567f2-244a-4f74-9b23-4b4949b6ac10-kube-api-access-qfcsr" pod "66f567f2-244a-4f74-9b23-4b4949b6ac10" (UID: "66f567f2-244a-4f74-9b23-4b4949b6ac10"). InnerVolumeSpecName "kube-api-access-qfcsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:33.814358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.814280    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:33.814358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.814313    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66f567f2-244a-4f74-9b23-4b4949b6ac10-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:33.814358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:33.814330    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcsr\" (UniqueName: \"kubernetes.io/projected/66f567f2-244a-4f74-9b23-4b4949b6ac10-kube-api-access-qfcsr\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:34.290305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:34.290271    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"
Nov 01 22:39:34.290502 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:34.290302    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt" event={"ID":"66f567f2-244a-4f74-9b23-4b4949b6ac10","Type":"ContainerDied","Data":"fac5b2f428fb505e52573ec1388bd32ba963a3aa844b6b48e53e6df646b3002e"}
Nov 01 22:39:34.290502 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:34.290351    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac5b2f428fb505e52573ec1388bd32ba963a3aa844b6b48e53e6df646b3002e"
Nov 01 22:39:42.322465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322431    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"]
Nov 01 22:39:42.322937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322923    4837 cpu_manager.go:414] "containerMap: removing container" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy"
Nov 01 22:39:42.322937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322938    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy"
Nov 01 22:39:42.323025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322948    4837 cpu_manager.go:414] "containerMap: removing container" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy"
Nov 01 22:39:42.323025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322953    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy"
Nov 01 22:39:42.323025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322964    4837 cpu_manager.go:414] "containerMap: removing container" podUID="66f567f2-244a-4f74-9b23-4b4949b6ac10" containerName="revoke-keys"
Nov 01 22:39:42.323025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322969    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f567f2-244a-4f74-9b23-4b4949b6ac10" containerName="revoke-keys"
Nov 01 22:39:42.323025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322990    4837 cpu_manager.go:414] "containerMap: removing container" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy"
Nov 01 22:39:42.323025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.322994    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy"
Nov 01 22:39:42.323235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.323080    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="535a603e-c963-4fc8-a816-2e1b8209120e" containerName="istio-proxy"
Nov 01 22:39:42.323235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.323088    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="832a757e-f762-446b-aa18-5425bad1c560" containerName="istio-proxy"
Nov 01 22:39:42.323235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.323093    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f567f2-244a-4f74-9b23-4b4949b6ac10" containerName="revoke-keys"
Nov 01 22:39:42.323235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.323102    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="551392b2-5288-4eda-8e42-3bf42fb8b7d2" containerName="istio-proxy"
Nov 01 22:39:42.327816 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.327797    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.379862 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.379829    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"]
Nov 01 22:39:42.400150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.400126    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.400360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.400333    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.400413 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.400389    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscjc\" (UniqueName: \"kubernetes.io/projected/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-kube-api-access-nscjc\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.422951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.422930    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:39:42.423050 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.422949    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:39:42.471507 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.471485    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"]
Nov 01 22:39:42.475321 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.475305    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.501646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.501625    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.501743 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.501659    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.501743 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.501698    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.501843 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.501830    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscjc\" (UniqueName: \"kubernetes.io/projected/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-kube-api-access-nscjc\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.501885 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.501872    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.501927 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.501899    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4qd\" (UniqueName: \"kubernetes.io/projected/e1966a5f-6cff-4108-bd27-01c55d399c07-kube-api-access-mh4qd\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.502578 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.502558    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.503909 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.503893    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.530751 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.530728    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"]
Nov 01 22:39:42.555468 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.555445    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscjc\" (UniqueName: \"kubernetes.io/projected/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-kube-api-access-nscjc\") pod \"maas-api-revoke-sub-80624021-6qs7t\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.602919 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.602862    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4qd\" (UniqueName: \"kubernetes.io/projected/e1966a5f-6cff-4108-bd27-01c55d399c07-kube-api-access-mh4qd\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.603018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.602925    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.603097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.603066    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.603596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.603576    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.605301 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.605279    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.642204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.642182    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:42.658166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.658141    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4qd\" (UniqueName: \"kubernetes.io/projected/e1966a5f-6cff-4108-bd27-01c55d399c07-kube-api-access-mh4qd\") pod \"maas-api-revoke-sub-16bbd381-fnbsz\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:42.789153 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:42.789127    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:43.010896 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:43.010867    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"]
Nov 01 22:39:43.012067 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:39:43.012026    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c49865f_a9ae_4c21_99cc_2ebf21e618aa.slice/crio-29564507575beb462cd50bb20b0d7fe200beda9d2bd81765a9f5845ecd47395a WatchSource:0}: Error finding container 29564507575beb462cd50bb20b0d7fe200beda9d2bd81765a9f5845ecd47395a: Status 404 returned error can't find the container with id 29564507575beb462cd50bb20b0d7fe200beda9d2bd81765a9f5845ecd47395a
Nov 01 22:39:43.158125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:43.158098    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"]
Nov 01 22:39:43.159767 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:39:43.159738    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1966a5f_6cff_4108_bd27_01c55d399c07.slice/crio-8623c923e91399e62213b7297eaba4b3c3c4966f3744d8488e578221f48afd3d WatchSource:0}: Error finding container 8623c923e91399e62213b7297eaba4b3c3c4966f3744d8488e578221f48afd3d: Status 404 returned error can't find the container with id 8623c923e91399e62213b7297eaba4b3c3c4966f3744d8488e578221f48afd3d
Nov 01 22:39:43.340891 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:43.340856    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz" event={"ID":"e1966a5f-6cff-4108-bd27-01c55d399c07","Type":"ContainerStarted","Data":"8623c923e91399e62213b7297eaba4b3c3c4966f3744d8488e578221f48afd3d"}
Nov 01 22:39:43.342605 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:43.342576    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t" event={"ID":"3c49865f-a9ae-4c21-99cc-2ebf21e618aa","Type":"ContainerStarted","Data":"29564507575beb462cd50bb20b0d7fe200beda9d2bd81765a9f5845ecd47395a"}
Nov 01 22:39:44.350036 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:44.350000    4837 generic.go:356] "Generic (PLEG): container finished" podID="e1966a5f-6cff-4108-bd27-01c55d399c07" containerID="826b9629a7e1b4f0c1cc98a07ef558e6c93973aaedb2604734df269d667a2ca4" exitCode=0
Nov 01 22:39:44.350458 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:44.350100    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz" event={"ID":"e1966a5f-6cff-4108-bd27-01c55d399c07","Type":"ContainerDied","Data":"826b9629a7e1b4f0c1cc98a07ef558e6c93973aaedb2604734df269d667a2ca4"}
Nov 01 22:39:44.351637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:44.351608    4837 generic.go:356] "Generic (PLEG): container finished" podID="3c49865f-a9ae-4c21-99cc-2ebf21e618aa" containerID="9548bf760663736afaa6a323f81bd9230604868fb1461a0e12a9c19851f4784a" exitCode=0
Nov 01 22:39:44.351741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:44.351671    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t" event={"ID":"3c49865f-a9ae-4c21-99cc-2ebf21e618aa","Type":"ContainerDied","Data":"9548bf760663736afaa6a323f81bd9230604868fb1461a0e12a9c19851f4784a"}
Nov 01 22:39:45.529936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.529908    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:45.555244 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.555220    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:45.638004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.637934    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1966a5f-6cff-4108-bd27-01c55d399c07-kube-api-access-mh4qd\" (UniqueName: \"kubernetes.io/projected/e1966a5f-6cff-4108-bd27-01c55d399c07-kube-api-access-mh4qd\") pod \"e1966a5f-6cff-4108-bd27-01c55d399c07\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") "
Nov 01 22:39:45.638163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638066    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-kube-api-access-nscjc\" (UniqueName: \"kubernetes.io/projected/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-kube-api-access-nscjc\") pod \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") "
Nov 01 22:39:45.638163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638146    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-openshift-ca\") pod \"e1966a5f-6cff-4108-bd27-01c55d399c07\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") "
Nov 01 22:39:45.638279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638171    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-xks-ca\") pod \"e1966a5f-6cff-4108-bd27-01c55d399c07\" (UID: \"e1966a5f-6cff-4108-bd27-01c55d399c07\") "
Nov 01 22:39:45.638279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638194    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-xks-ca\") pod \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") "
Nov 01 22:39:45.638279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638252    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-openshift-ca\") pod \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\" (UID: \"3c49865f-a9ae-4c21-99cc-2ebf21e618aa\") "
Nov 01 22:39:45.638581 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638555    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-openshift-ca" pod "e1966a5f-6cff-4108-bd27-01c55d399c07" (UID: "e1966a5f-6cff-4108-bd27-01c55d399c07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:45.638680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638634    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-openshift-ca" pod "3c49865f-a9ae-4c21-99cc-2ebf21e618aa" (UID: "3c49865f-a9ae-4c21-99cc-2ebf21e618aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:45.639064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638798    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:45.639064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.638824    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:45.640921 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.640896    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1966a5f-6cff-4108-bd27-01c55d399c07-kube-api-access-mh4qd" pod "e1966a5f-6cff-4108-bd27-01c55d399c07" (UID: "e1966a5f-6cff-4108-bd27-01c55d399c07"). InnerVolumeSpecName "kube-api-access-mh4qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:45.641159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.641139    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-kube-api-access-nscjc" pod "3c49865f-a9ae-4c21-99cc-2ebf21e618aa" (UID: "3c49865f-a9ae-4c21-99cc-2ebf21e618aa"). InnerVolumeSpecName "kube-api-access-nscjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:45.641236 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.641163    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-xks-ca" pod "3c49865f-a9ae-4c21-99cc-2ebf21e618aa" (UID: "3c49865f-a9ae-4c21-99cc-2ebf21e618aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:45.641295 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.641263    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-xks-ca" pod "e1966a5f-6cff-4108-bd27-01c55d399c07" (UID: "e1966a5f-6cff-4108-bd27-01c55d399c07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:45.739738 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.739712    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscjc\" (UniqueName: \"kubernetes.io/projected/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-kube-api-access-nscjc\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:45.739738 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.739736    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1966a5f-6cff-4108-bd27-01c55d399c07-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:45.739879 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.739746    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c49865f-a9ae-4c21-99cc-2ebf21e618aa-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:45.739879 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:45.739755    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh4qd\" (UniqueName: \"kubernetes.io/projected/e1966a5f-6cff-4108-bd27-01c55d399c07-kube-api-access-mh4qd\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:46.364764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:46.364736    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"
Nov 01 22:39:46.365029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:46.364777    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz" event={"ID":"e1966a5f-6cff-4108-bd27-01c55d399c07","Type":"ContainerDied","Data":"8623c923e91399e62213b7297eaba4b3c3c4966f3744d8488e578221f48afd3d"}
Nov 01 22:39:46.365029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:46.364851    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8623c923e91399e62213b7297eaba4b3c3c4966f3744d8488e578221f48afd3d"
Nov 01 22:39:46.366163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:46.366146    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"
Nov 01 22:39:46.366261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:46.366165    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t" event={"ID":"3c49865f-a9ae-4c21-99cc-2ebf21e618aa","Type":"ContainerDied","Data":"29564507575beb462cd50bb20b0d7fe200beda9d2bd81765a9f5845ecd47395a"}
Nov 01 22:39:46.366261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:46.366205    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29564507575beb462cd50bb20b0d7fe200beda9d2bd81765a9f5845ecd47395a"
Nov 01 22:39:48.811859 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:39:48.811827    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:39:53.231637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.231599    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"]
Nov 01 22:39:53.232143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.232129    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3c49865f-a9ae-4c21-99cc-2ebf21e618aa" containerName="revoke-keys"
Nov 01 22:39:53.232195 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.232146    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c49865f-a9ae-4c21-99cc-2ebf21e618aa" containerName="revoke-keys"
Nov 01 22:39:53.232195 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.232185    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e1966a5f-6cff-4108-bd27-01c55d399c07" containerName="revoke-keys"
Nov 01 22:39:53.232195 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.232191    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1966a5f-6cff-4108-bd27-01c55d399c07" containerName="revoke-keys"
Nov 01 22:39:53.232316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.232278    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1966a5f-6cff-4108-bd27-01c55d399c07" containerName="revoke-keys"
Nov 01 22:39:53.232316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.232295    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c49865f-a9ae-4c21-99cc-2ebf21e618aa" containerName="revoke-keys"
Nov 01 22:39:53.235618 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.235599    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.284982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.284956    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:39:53.314555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.314523    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.314680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.314567    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8qtr\" (UniqueName: \"kubernetes.io/projected/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-kube-api-access-h8qtr\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.314791 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.314771    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.331314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.331296    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:39:53.333636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.333618    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"]
Nov 01 22:39:53.415507 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.415479    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.415627 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.415522    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8qtr\" (UniqueName: \"kubernetes.io/projected/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-kube-api-access-h8qtr\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.415718 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.415702    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.416361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.416340    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.417742 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.417723    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.470449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.470424    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8qtr\" (UniqueName: \"kubernetes.io/projected/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-kube-api-access-h8qtr\") pod \"maas-api-revoke-sub-75ab4a11-kdc62\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.550118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.550063    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:53.923469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:53.923442    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"]
Nov 01 22:39:53.924601 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:39:53.924555    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e799b3_e11d_4d64_ab5c_328e2abb44fd.slice/crio-3356a7008bc53361f877714e3a2e3afd1ea4fd2395612bcc863aa400941bf759 WatchSource:0}: Error finding container 3356a7008bc53361f877714e3a2e3afd1ea4fd2395612bcc863aa400941bf759: Status 404 returned error can't find the container with id 3356a7008bc53361f877714e3a2e3afd1ea4fd2395612bcc863aa400941bf759
Nov 01 22:39:54.409704 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.409663    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62" event={"ID":"f3e799b3-e11d-4d64-ab5c-328e2abb44fd","Type":"ContainerStarted","Data":"1414211422dc9f0bdcae384f5220f26b2ebb8dd25592ba52cfe1f0f7c1db7a46"}
Nov 01 22:39:54.410083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.409718    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62" event={"ID":"f3e799b3-e11d-4d64-ab5c-328e2abb44fd","Type":"ContainerStarted","Data":"3356a7008bc53361f877714e3a2e3afd1ea4fd2395612bcc863aa400941bf759"}
Nov 01 22:39:54.529497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.524895    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62" podStartSLOduration=1.524867625 podStartE2EDuration="1.524867625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:54.522825448 +0000 UTC m=+976.112024510" watchObservedRunningTime="2026-11-01 22:39:54.524867625 +0000 UTC m=+976.114066686"
Nov 01 22:39:54.853461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.853429    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"]
Nov 01 22:39:54.857297 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.857270    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.874760 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.874734    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"]
Nov 01 22:39:54.878362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.878345    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:54.900104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.900080    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"]
Nov 01 22:39:54.903974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.903959    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:54.923171 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.923147    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"]
Nov 01 22:39:54.931706 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.931682    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.931866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.931736    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9xr5\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-kube-api-access-z9xr5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.931866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.931804    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.932491 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.932466    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1a58aa0-2864-43d4-b7be-60228b973837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.932593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.932505    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.932593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.932529    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.932593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.932561    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.932756 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.932667    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.932756 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.932702    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1a58aa0-2864-43d4-b7be-60228b973837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:54.965577 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.965553    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"]
Nov 01 22:39:54.979512 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:54.979486    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"]
Nov 01 22:39:55.033940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.033902    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.034138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034087    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1a58aa0-2864-43d4-b7be-60228b973837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.034213 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034132    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndhn\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-kube-api-access-bndhn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.034213 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034179    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.034213 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034204    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034245    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034272    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034317    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034356    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034388    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034425    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034470    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034501    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.034558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034527    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034583    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034623    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034650    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfcadb-8c03-4339-8129-7814d204dfe2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034677    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1a58aa0-2864-43d4-b7be-60228b973837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034718    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034751    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034784    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p84xg\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-kube-api-access-p84xg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034826    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034865    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034921    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9xr5\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-kube-api-access-z9xr5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.034988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.034975    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.035559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.035008    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.035559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.035038    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.035559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.035472    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.040079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.036249    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.040079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.036517    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.040079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.036737    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.040079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.038701    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1a58aa0-2864-43d4-b7be-60228b973837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.040079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.039725    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.040079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.040035    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1a58aa0-2864-43d4-b7be-60228b973837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.089404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.089347    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.090922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.090896    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9xr5\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-kube-api-access-z9xr5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cp2r6\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.106675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.106651    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:39:55.136152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136128    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.136241 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136158    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p84xg\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-kube-api-access-p84xg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.136241 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136186    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.136241 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136222    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.136380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136241    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.136380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136271    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.136380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136299    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndhn\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-kube-api-access-bndhn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.136380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136324    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.136380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136357    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.136634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136392    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.136634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136423    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.136634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136480    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.136634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136496    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.136634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136518    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.136634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136546    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.136634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136571    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.136634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136603    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.137079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136646    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.137079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136676    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfcadb-8c03-4339-8129-7814d204dfe2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.137079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136716    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.137079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136728    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.137079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136763    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.137079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.136993    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.137079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.137014    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.137448 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.137127    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.137448 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.137187    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.137448 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.137347    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.137863 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.137837    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfcadb-8c03-4339-8129-7814d204dfe2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.138954 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.138931    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.139094 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.139070    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.139296 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.139281    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.139358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.139342    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.190033 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.190004    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p84xg\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-kube-api-access-p84xg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.190184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.190073    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tm4js\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.190993 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.190975    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.191192 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.191176    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndhn\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-kube-api-access-bndhn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ggm2j\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.219718 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.219696    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:55.229346 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.229328    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:55.242037 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.242015    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:55.461121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.461073    4837 generic.go:356] "Generic (PLEG): container finished" podID="f3e799b3-e11d-4d64-ab5c-328e2abb44fd" containerID="1414211422dc9f0bdcae384f5220f26b2ebb8dd25592ba52cfe1f0f7c1db7a46" exitCode=0
Nov 01 22:39:55.461555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.461221    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62" event={"ID":"f3e799b3-e11d-4d64-ab5c-328e2abb44fd","Type":"ContainerDied","Data":"1414211422dc9f0bdcae384f5220f26b2ebb8dd25592ba52cfe1f0f7c1db7a46"}
Nov 01 22:39:55.684068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.678827    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"]
Nov 01 22:39:55.691102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.688444    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"]
Nov 01 22:39:55.706066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.700137    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"]
Nov 01 22:39:55.717405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.708993    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:55.717405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.709103    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:55.717405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.709142    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:55.721104 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:39:55.719506    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07815f2_3a45_405c_a1cc_1a1f703c0e5c.slice/crio-01f9b59edb034afbea99420845d000d2f114d15cc4e3ebc64991e42038dce798 WatchSource:0}: Error finding container 01f9b59edb034afbea99420845d000d2f114d15cc4e3ebc64991e42038dce798: Status 404 returned error can't find the container with id 01f9b59edb034afbea99420845d000d2f114d15cc4e3ebc64991e42038dce798
Nov 01 22:39:55.721104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.720145    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:55.721104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.720219    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:55.721104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.720263    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:55.726802 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.726768    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:55.726912 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.726846    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:55.726912 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:55.726907    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:39:56.612836 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:56.612101    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" event={"ID":"f07815f2-3a45-405c-a1cc-1a1f703c0e5c","Type":"ContainerStarted","Data":"86969515dc73b4da914bbe1108764e9072e421cfea4f3de89873dbc7055ca8e3"}
Nov 01 22:39:56.612836 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:56.612174    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" event={"ID":"f07815f2-3a45-405c-a1cc-1a1f703c0e5c","Type":"ContainerStarted","Data":"01f9b59edb034afbea99420845d000d2f114d15cc4e3ebc64991e42038dce798"}
Nov 01 22:39:56.616501 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:56.616468    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" event={"ID":"cabfcadb-8c03-4339-8129-7814d204dfe2","Type":"ContainerStarted","Data":"d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f"}
Nov 01 22:39:56.616787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:56.616748    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" event={"ID":"cabfcadb-8c03-4339-8129-7814d204dfe2","Type":"ContainerStarted","Data":"a300cd0f8c2fc00f45735cf2e176d194adc292fffa58c6228f10112b50a58f45"}
Nov 01 22:39:56.620688 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:56.620600    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" event={"ID":"c1a58aa0-2864-43d4-b7be-60228b973837","Type":"ContainerStarted","Data":"00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4"}
Nov 01 22:39:56.620688 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:56.620639    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" event={"ID":"c1a58aa0-2864-43d4-b7be-60228b973837","Type":"ContainerStarted","Data":"f3afaa7b6c6f90d261f533d29c3ebcdadb3616cd6964786fbc01cdc0ad606805"}
Nov 01 22:39:57.139959 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.139895    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podStartSLOduration=3.139878135 podStartE2EDuration="3.139878135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:57.138422459 +0000 UTC m=+978.727621521" watchObservedRunningTime="2026-11-01 22:39:57.139878135 +0000 UTC m=+978.729077197"
Nov 01 22:39:57.224481 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.224393    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:39:57.225587 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.225538    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podStartSLOduration=3.225525678 podStartE2EDuration="3.225525678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:56.878799272 +0000 UTC m=+978.467998373" watchObservedRunningTime="2026-11-01 22:39:57.225525678 +0000 UTC m=+978.814724740"
Nov 01 22:39:57.230359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.230340    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:39:57.260090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.251951    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:39:57.301251 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.301196    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:39:57.301591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.301556    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:39:57.307389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.307355    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:39:57.311177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.311138    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:39:57.324984 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.313076    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:39:57.591243 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.591203    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 22:39:57.591550 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.587506    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:57.591950 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.354847    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podStartSLOduration=3.354824984 podStartE2EDuration="3.354824984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:57.298185916 +0000 UTC m=+978.887384978" watchObservedRunningTime="2026-11-01 22:39:57.354824984 +0000 UTC m=+978.944024049"
Nov 01 22:39:57.688461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.688405    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-xks-ca\") pod \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") "
Nov 01 22:39:57.700245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.700209    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-openshift-ca\") pod \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") "
Nov 01 22:39:57.700594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.700575    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-kube-api-access-h8qtr\" (UniqueName: \"kubernetes.io/projected/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-kube-api-access-h8qtr\") pod \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\" (UID: \"f3e799b3-e11d-4d64-ab5c-328e2abb44fd\") "
Nov 01 22:39:57.712066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.701413    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-openshift-ca" pod "f3e799b3-e11d-4d64-ab5c-328e2abb44fd" (UID: "f3e799b3-e11d-4d64-ab5c-328e2abb44fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:57.731686 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.731654    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-xks-ca" pod "f3e799b3-e11d-4d64-ab5c-328e2abb44fd" (UID: "f3e799b3-e11d-4d64-ab5c-328e2abb44fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:57.744506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.742786    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"
Nov 01 22:39:57.744506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.742916    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62" event={"ID":"f3e799b3-e11d-4d64-ab5c-328e2abb44fd","Type":"ContainerDied","Data":"3356a7008bc53361f877714e3a2e3afd1ea4fd2395612bcc863aa400941bf759"}
Nov 01 22:39:57.744506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.742959    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3356a7008bc53361f877714e3a2e3afd1ea4fd2395612bcc863aa400941bf759"
Nov 01 22:39:57.745616 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.745594    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:57.752499 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.752476    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:57.757508 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.757459    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-kube-api-access-h8qtr" pod "f3e799b3-e11d-4d64-ab5c-328e2abb44fd" (UID: "f3e799b3-e11d-4d64-ab5c-328e2abb44fd"). InnerVolumeSpecName "kube-api-access-h8qtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:57.880107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:57.857373    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8qtr\" (UniqueName: \"kubernetes.io/projected/f3e799b3-e11d-4d64-ab5c-328e2abb44fd-kube-api-access-h8qtr\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:58.117126 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.113501    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:39:58.220105 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.215303    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:39:58.224412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.224376    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:39:58.224591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.224434    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 22:39:58.232528 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.232385    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:39:58.232528 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.232460    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:39:58.245574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.245364    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:39:58.245574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.245417    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:39:58.394157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.394013    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"]
Nov 01 22:39:58.398075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.394803    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f3e799b3-e11d-4d64-ab5c-328e2abb44fd" containerName="revoke-keys"
Nov 01 22:39:58.398075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.394826    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e799b3-e11d-4d64-ab5c-328e2abb44fd" containerName="revoke-keys"
Nov 01 22:39:58.398075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.394989    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e799b3-e11d-4d64-ab5c-328e2abb44fd" containerName="revoke-keys"
Nov 01 22:39:58.411847 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.411823    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:58.488862 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.488833    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:39:58.489717 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.489698    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:39:58.491536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.491503    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"]
Nov 01 22:39:58.574187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.574149    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:58.623271 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.623238    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rfbb\" (UniqueName: \"kubernetes.io/projected/4c7ad130-4925-4b98-a80a-0215770e39a8-kube-api-access-4rfbb\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:58.623860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.623837    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:58.756100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.728012    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rfbb\" (UniqueName: \"kubernetes.io/projected/4c7ad130-4925-4b98-a80a-0215770e39a8-kube-api-access-4rfbb\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:58.773308 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.773266    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:58.911536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.911503    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:58.912305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.889035    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rfbb\" (UniqueName: \"kubernetes.io/projected/4c7ad130-4925-4b98-a80a-0215770e39a8-kube-api-access-4rfbb\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:58.912470 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:58.818573    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:59.030128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.008571    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657793f-p2rsn\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:59.060240 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.060200    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:39:59.110674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.108178    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 01 22:39:59.110674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.108510    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:39:59.115058 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.115018    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:39:59.149067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.127279    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:39:59.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.225376    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:39:59.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.225432    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 22:39:59.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.225502    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:39:59.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.225799    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:39:59.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.238357    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:39:59.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.238404    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:39:59.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.244317    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:39:59.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.244360    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:39:59.486206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.486163    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:39:59.557094 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.551857    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:39:59.880077 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:39:59.860218    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:40:00.410561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:00.401159    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:40:00.410561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:00.401240    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 22:40:00.410561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:00.401749    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:40:00.410561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:00.401803    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:40:00.535067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:00.534253    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:40:00.535067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:00.534310    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:40:00.810068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:00.792249    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:40:01.221870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:01.221832    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:40:01.222240 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:01.221908    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 22:40:01.264076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:01.238138    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:40:01.264076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:01.238195    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:40:01.304083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:01.298663    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:40:01.304083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:01.298724    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:40:01.410232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:01.409582    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"]
Nov 01 22:40:02.182957 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.181859    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn" event={"ID":"4c7ad130-4925-4b98-a80a-0215770e39a8","Type":"ContainerStarted","Data":"a0f9001328d94399ce1ed3a677ebc318874c16002d76f9836334d98920fe3c57"}
Nov 01 22:40:02.182957 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.181935    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn" event={"ID":"4c7ad130-4925-4b98-a80a-0215770e39a8","Type":"ContainerStarted","Data":"14ee5dbb1fff56e96772fbd3c79adedf0f3791e64bde8864e883497f3474bfd3"}
Nov 01 22:40:02.220448 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.220411    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:40:02.220601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.220464    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 22:40:02.229891 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.229866    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:40:02.230263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.229907    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:40:02.247112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.244584    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:40:02.247112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.244648    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:40:02.310069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:02.309423    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn" podStartSLOduration=4.309402524 podStartE2EDuration="4.309402524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:02.308322778 +0000 UTC m=+983.897521840" watchObservedRunningTime="2026-11-01 22:40:02.309402524 +0000 UTC m=+983.898601588"
Nov 01 22:40:03.212494 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:03.212454    4837 generic.go:356] "Generic (PLEG): container finished" podID="4c7ad130-4925-4b98-a80a-0215770e39a8" containerID="a0f9001328d94399ce1ed3a677ebc318874c16002d76f9836334d98920fe3c57" exitCode=0
Nov 01 22:40:03.212662 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:03.212536    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn" event={"ID":"4c7ad130-4925-4b98-a80a-0215770e39a8","Type":"ContainerDied","Data":"a0f9001328d94399ce1ed3a677ebc318874c16002d76f9836334d98920fe3c57"}
Nov 01 22:40:03.220747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:03.220727    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:40:03.220819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:03.220771    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 22:40:03.234066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:03.230257    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:40:03.234066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:03.230329    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:40:03.243383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:03.243354    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:40:03.243581 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:03.243555    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:40:04.220090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.220036    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:40:04.220242 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.220117    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 22:40:04.230114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.230083    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:40:04.230231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.230154    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:40:04.242593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.242564    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:40:04.243095 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.242633    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:40:04.395847 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.395815    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:40:04.462377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.462026    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7ad130-4925-4b98-a80a-0215770e39a8-kube-api-access-4rfbb\" (UniqueName: \"kubernetes.io/projected/4c7ad130-4925-4b98-a80a-0215770e39a8-kube-api-access-4rfbb\") pod \"4c7ad130-4925-4b98-a80a-0215770e39a8\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") "
Nov 01 22:40:04.462377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.462124    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-openshift-ca\") pod \"4c7ad130-4925-4b98-a80a-0215770e39a8\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") "
Nov 01 22:40:04.462377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.462183    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-xks-ca\") pod \"4c7ad130-4925-4b98-a80a-0215770e39a8\" (UID: \"4c7ad130-4925-4b98-a80a-0215770e39a8\") "
Nov 01 22:40:04.462685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.462565    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-openshift-ca" pod "4c7ad130-4925-4b98-a80a-0215770e39a8" (UID: "4c7ad130-4925-4b98-a80a-0215770e39a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:40:04.462795 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.462774    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:04.470175 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.465021    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-xks-ca" pod "4c7ad130-4925-4b98-a80a-0215770e39a8" (UID: "4c7ad130-4925-4b98-a80a-0215770e39a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:40:04.470659 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.470482    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7ad130-4925-4b98-a80a-0215770e39a8-kube-api-access-4rfbb" pod "4c7ad130-4925-4b98-a80a-0215770e39a8" (UID: "4c7ad130-4925-4b98-a80a-0215770e39a8"). InnerVolumeSpecName "kube-api-access-4rfbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:40:04.564241 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.564187    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rfbb\" (UniqueName: \"kubernetes.io/projected/4c7ad130-4925-4b98-a80a-0215770e39a8-kube-api-access-4rfbb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:04.564363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:04.564246    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c7ad130-4925-4b98-a80a-0215770e39a8-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:05.229948 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:05.229908    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tm4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 22:40:05.230160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:05.229969    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 22:40:05.231541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:05.231513    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn" event={"ID":"4c7ad130-4925-4b98-a80a-0215770e39a8","Type":"ContainerDied","Data":"14ee5dbb1fff56e96772fbd3c79adedf0f3791e64bde8864e883497f3474bfd3"}
Nov 01 22:40:05.231942 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:05.231906    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14ee5dbb1fff56e96772fbd3c79adedf0f3791e64bde8864e883497f3474bfd3"
Nov 01 22:40:05.231942 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:05.231528    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"
Nov 01 22:40:05.242826 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:05.242800    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ggm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:40:05.243255 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:05.242852    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 22:40:06.243562 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.240592    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cp2r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:40:06.243562 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.240654    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:40:06.282653 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.282617    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:40:06.283105 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.283078    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:40:06.284392 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.284370    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:40:06.499803 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.499727    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:40:06.500293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.500268    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:40:06.501126 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.501109    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:40:06.594160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.594130    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:40:06.594527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.594505    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:40:06.595291 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:06.595273    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:40:09.243443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.243408    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"]
Nov 01 22:40:09.243951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.243936    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4c7ad130-4925-4b98-a80a-0215770e39a8" containerName="revoke-keys"
Nov 01 22:40:09.243951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.243950    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7ad130-4925-4b98-a80a-0215770e39a8" containerName="revoke-keys"
Nov 01 22:40:09.244075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.244029    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7ad130-4925-4b98-a80a-0215770e39a8" containerName="revoke-keys"
Nov 01 22:40:09.250943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.250922    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.304433 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.304402    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"]
Nov 01 22:40:09.320097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.320069    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.320218 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.320102    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l22v8\" (UniqueName: \"kubernetes.io/projected/18979efd-25a5-4f49-97f0-fef4604faae1-kube-api-access-l22v8\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.320330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.320317    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.346615 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.346584    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:40:09.346749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.346643    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:40:09.421157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.421120    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.421328 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.421298    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.421328 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.421324    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22v8\" (UniqueName: \"kubernetes.io/projected/18979efd-25a5-4f49-97f0-fef4604faae1-kube-api-access-l22v8\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.421912 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.421895    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.423401 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.423375    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.475794 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.475763    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l22v8\" (UniqueName: \"kubernetes.io/projected/18979efd-25a5-4f49-97f0-fef4604faae1-kube-api-access-l22v8\") pod \"maas-api-revoke-sub-a7bb9326-rcvkf\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.564888 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.564830    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:09.931861 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:09.931828    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"]
Nov 01 22:40:09.932015 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:40:09.931994    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18979efd_25a5_4f49_97f0_fef4604faae1.slice/crio-7c72b9f81c8401b4d0b88a0691e893973c0ef5c4e5624515f98c2a288b812cf1 WatchSource:0}: Error finding container 7c72b9f81c8401b4d0b88a0691e893973c0ef5c4e5624515f98c2a288b812cf1: Status 404 returned error can't find the container with id 7c72b9f81c8401b4d0b88a0691e893973c0ef5c4e5624515f98c2a288b812cf1
Nov 01 22:40:10.272449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:10.272416    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf" event={"ID":"18979efd-25a5-4f49-97f0-fef4604faae1","Type":"ContainerStarted","Data":"172b16c90ef6164a532d00ae357919cd51a0af4b2a7b0f0def7eb0e5fd0ca76b"}
Nov 01 22:40:10.272754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:10.272478    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf" event={"ID":"18979efd-25a5-4f49-97f0-fef4604faae1","Type":"ContainerStarted","Data":"7c72b9f81c8401b4d0b88a0691e893973c0ef5c4e5624515f98c2a288b812cf1"}
Nov 01 22:40:10.380415 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:10.380371    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf" podStartSLOduration=1.380359277 podStartE2EDuration="1.380359277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:10.380156378 +0000 UTC m=+991.969355439" watchObservedRunningTime="2026-11-01 22:40:10.380359277 +0000 UTC m=+991.969558338"
Nov 01 22:40:11.279697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:11.279667    4837 generic.go:356] "Generic (PLEG): container finished" podID="18979efd-25a5-4f49-97f0-fef4604faae1" containerID="172b16c90ef6164a532d00ae357919cd51a0af4b2a7b0f0def7eb0e5fd0ca76b" exitCode=0
Nov 01 22:40:11.280095 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:11.279711    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf" event={"ID":"18979efd-25a5-4f49-97f0-fef4604faae1","Type":"ContainerDied","Data":"172b16c90ef6164a532d00ae357919cd51a0af4b2a7b0f0def7eb0e5fd0ca76b"}
Nov 01 22:40:12.435402 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.435373    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:12.555397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.555325    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-xks-ca\") pod \"18979efd-25a5-4f49-97f0-fef4604faae1\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") "
Nov 01 22:40:12.555525 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.555394    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-openshift-ca\") pod \"18979efd-25a5-4f49-97f0-fef4604faae1\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") "
Nov 01 22:40:12.555584 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.555529    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18979efd-25a5-4f49-97f0-fef4604faae1-kube-api-access-l22v8\" (UniqueName: \"kubernetes.io/projected/18979efd-25a5-4f49-97f0-fef4604faae1-kube-api-access-l22v8\") pod \"18979efd-25a5-4f49-97f0-fef4604faae1\" (UID: \"18979efd-25a5-4f49-97f0-fef4604faae1\") "
Nov 01 22:40:12.555822 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.555784    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-openshift-ca" pod "18979efd-25a5-4f49-97f0-fef4604faae1" (UID: "18979efd-25a5-4f49-97f0-fef4604faae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:40:12.558098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.558069    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18979efd-25a5-4f49-97f0-fef4604faae1-kube-api-access-l22v8" pod "18979efd-25a5-4f49-97f0-fef4604faae1" (UID: "18979efd-25a5-4f49-97f0-fef4604faae1"). InnerVolumeSpecName "kube-api-access-l22v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:40:12.558310 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.558283    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-xks-ca" pod "18979efd-25a5-4f49-97f0-fef4604faae1" (UID: "18979efd-25a5-4f49-97f0-fef4604faae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:40:12.656687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.656657    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:12.656801 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.656691    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18979efd-25a5-4f49-97f0-fef4604faae1-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:12.656801 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:12.656708    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l22v8\" (UniqueName: \"kubernetes.io/projected/18979efd-25a5-4f49-97f0-fef4604faae1-kube-api-access-l22v8\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:13.292111 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:13.292075    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf" event={"ID":"18979efd-25a5-4f49-97f0-fef4604faae1","Type":"ContainerDied","Data":"7c72b9f81c8401b4d0b88a0691e893973c0ef5c4e5624515f98c2a288b812cf1"}
Nov 01 22:40:13.292274 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:13.292131    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c72b9f81c8401b4d0b88a0691e893973c0ef5c4e5624515f98c2a288b812cf1"
Nov 01 22:40:13.292274 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:13.292086    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"
Nov 01 22:40:18.813324 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:40:18.813288    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:40:24.966096 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:24.966065    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"]
Nov 01 22:40:24.966684 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:24.966656    4837 cpu_manager.go:414] "containerMap: removing container" podUID="18979efd-25a5-4f49-97f0-fef4604faae1" containerName="revoke-keys"
Nov 01 22:40:24.966684 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:24.966676    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18979efd-25a5-4f49-97f0-fef4604faae1" containerName="revoke-keys"
Nov 01 22:40:24.966974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:24.966786    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="18979efd-25a5-4f49-97f0-fef4604faae1" containerName="revoke-keys"
Nov 01 22:40:24.970470 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:24.970451    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.020576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.020546    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-d6101b-openshift-default-dockercfg-jbnc7\""
Nov 01 22:40:25.029974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.029950    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"]
Nov 01 22:40:25.087660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.087634    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.087660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.087664    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583275c8-2239-4b6e-9236-307cea25f813-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.087862 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.087696    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.087862 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.087766    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583275c8-2239-4b6e-9236-307cea25f813-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.087945 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.087900    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.087979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.087954    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.088016 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.087977    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.088123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.088103    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl6gf\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-kube-api-access-nl6gf\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.088235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.088220    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.189676 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189651    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.189676 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189680    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583275c8-2239-4b6e-9236-307cea25f813-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.189869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189709    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.189869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189729    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583275c8-2239-4b6e-9236-307cea25f813-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.189869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189759    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.189869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189802    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.189869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189824    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.190185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189889    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl6gf\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-kube-api-access-nl6gf\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.190185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.189968    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.190185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.190118    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.190185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.190136    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.190389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.190189    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.190389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.190322    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.190652 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.190635    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583275c8-2239-4b6e-9236-307cea25f813-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.192212 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.192190    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.192382 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.192365    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583275c8-2239-4b6e-9236-307cea25f813-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.244701 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.244646    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.244783 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.244706    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl6gf\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-kube-api-access-nl6gf\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.286370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.286347    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:25.750649 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.750621    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"]
Nov 01 22:40:25.751397 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:40:25.751372    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583275c8_2239_4b6e_9236_307cea25f813.slice/crio-a26ae1ded572860b9c79a8235c8eeea8e3ec244a13eb83af5b69c929d32bc7e7 WatchSource:0}: Error finding container a26ae1ded572860b9c79a8235c8eeea8e3ec244a13eb83af5b69c929d32bc7e7: Status 404 returned error can't find the container with id a26ae1ded572860b9c79a8235c8eeea8e3ec244a13eb83af5b69c929d32bc7e7
Nov 01 22:40:25.753703 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.753673    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:40:25.753794 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.753738    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:40:25.753794 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:25.753768    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:40:26.372418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:26.372379    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" event={"ID":"583275c8-2239-4b6e-9236-307cea25f813","Type":"ContainerStarted","Data":"7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4"}
Nov 01 22:40:26.372815 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:26.372444    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" event={"ID":"583275c8-2239-4b6e-9236-307cea25f813","Type":"ContainerStarted","Data":"a26ae1ded572860b9c79a8235c8eeea8e3ec244a13eb83af5b69c929d32bc7e7"}
Nov 01 22:40:27.286941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.286910    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:27.288604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.288582    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:40:27.288665 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.288629    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 22:40:27.583622 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.583529    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podStartSLOduration=3.583514692 podStartE2EDuration="3.583514692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:26.554481413 +0000 UTC m=+1008.143680485" watchObservedRunningTime="2026-11-01 22:40:27.583514692 +0000 UTC m=+1009.172713753"
Nov 01 22:40:27.584617 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.584599    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"]
Nov 01 22:40:27.588399 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.588379    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.650064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.639956    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c768f3-openshift-default-dockercfg-r664d\""
Nov 01 22:40:27.706413 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.706389    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"]
Nov 01 22:40:27.718912 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.718888    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.719088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.718944    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.719088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.719022    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.719215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.719117    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.719215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.719146    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.719215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.719203    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28eac005-a391-4ba5-b018-c870d18a1033-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.719377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.719233    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvf9x\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-kube-api-access-xvf9x\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.719377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.719260    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.719377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.719327    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28eac005-a391-4ba5-b018-c870d18a1033-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820341 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820313    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820362    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28eac005-a391-4ba5-b018-c870d18a1033-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820448    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820491    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820514    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820539    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820564    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820583    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28eac005-a391-4ba5-b018-c870d18a1033-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.820739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.820606    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvf9x\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-kube-api-access-xvf9x\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.821447 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.821418    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.821565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.821429    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.821565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.821553    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.821682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.821651    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.821974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.821954    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28eac005-a391-4ba5-b018-c870d18a1033-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.823319 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.823298    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.823694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.823676    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28eac005-a391-4ba5-b018-c870d18a1033-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.879613 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.879546    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.880911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.880889    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvf9x\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-kube-api-access-xvf9x\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:27.902707 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:27.902683    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:28.287552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:28.287519    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:40:28.287720 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:28.287588    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 22:40:28.360175 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:28.360135    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"]
Nov 01 22:40:28.389656 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:40:28.389628    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1 WatchSource:0}: Error finding container 26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1: Status 404 returned error can't find the container with id 26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1
Nov 01 22:40:28.391874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:28.391844    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:40:28.391976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:28.391924    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:40:28.392038 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:28.391971    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:40:29.286687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:29.286656    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:40:29.287110 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:29.286711    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 22:40:29.398031 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:29.397984    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" event={"ID":"28eac005-a391-4ba5-b018-c870d18a1033","Type":"ContainerStarted","Data":"42290030f6164fb3169243b25edefd856732539951637bf718ff94aa5a740378"}
Nov 01 22:40:29.398322 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:29.398104    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" event={"ID":"28eac005-a391-4ba5-b018-c870d18a1033","Type":"ContainerStarted","Data":"26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1"}
Nov 01 22:40:29.520068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:29.519986    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podStartSLOduration=2.519969926 podStartE2EDuration="2.519969926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:29.519447126 +0000 UTC m=+1011.108646187" watchObservedRunningTime="2026-11-01 22:40:29.519969926 +0000 UTC m=+1011.109168988"
Nov 01 22:40:29.903656 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:29.903624    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:29.905293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:29.905271    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:40:29.905382 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:29.905331    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 22:40:30.287307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:30.287268    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:40:30.287799 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:30.287347    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 22:40:30.903088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:30.903036    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:40:30.903272 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:30.903107    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 22:40:31.287343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:31.287310    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:40:31.287728 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:31.287371    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 22:40:31.903527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:31.903487    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:40:31.903725 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:31.903541    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 22:40:32.286933 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:32.286900    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:40:32.287113 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:32.286975    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 22:40:32.904033 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:32.903999    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:40:32.904479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:32.904092    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 22:40:33.291478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:33.291447    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:33.291846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:33.291817    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:33.292626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:33.292601    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:40:33.903146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:33.903113    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:40:33.903324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:33.903173    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 22:40:34.903235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:34.903202    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:40:34.903621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:34.903267    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 22:40:35.907376 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:35.907348    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:35.907804 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:35.907718    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:35.908380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:35.908361    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:40:37.700198 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:40:37.700168    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:40:48.812300 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:40:48.812214    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:41:06.541594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.541562    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"]
Nov 01 22:41:06.545395 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.545380    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.596845 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.596821    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"distinct-w3-e71d55-kserve-self-signed-certs\""
Nov 01 22:41:06.646021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.645993    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"]
Nov 01 22:41:06.693074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.682717    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-model-cache\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.693074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.682802    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-home\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.693074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.682868    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-tmp-dir\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.693074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.682905    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngln6\" (UniqueName: \"kubernetes.io/projected/c1acf4ec-d463-4380-be2b-2da98bf7d984-kube-api-access-ngln6\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.693074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.682944    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-dshm\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.693074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.683102    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1acf4ec-d463-4380-be2b-2da98bf7d984-tls-certs\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.784435 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784397    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-model-cache\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.784614 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784479    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-home\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.784614 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784535    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-tmp-dir\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.784614 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784572    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngln6\" (UniqueName: \"kubernetes.io/projected/c1acf4ec-d463-4380-be2b-2da98bf7d984-kube-api-access-ngln6\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.784614 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784610    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-dshm\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.784908 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784729    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1acf4ec-d463-4380-be2b-2da98bf7d984-tls-certs\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.784908 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784887    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-model-cache\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.785019 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784915    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-home\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.785019 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.784937    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-tmp-dir\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.787025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.786998    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-dshm\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.787321 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.787302    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1acf4ec-d463-4380-be2b-2da98bf7d984-tls-certs\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.837931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.837867    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngln6\" (UniqueName: \"kubernetes.io/projected/c1acf4ec-d463-4380-be2b-2da98bf7d984-kube-api-access-ngln6\") pod \"distinct-w3-e71d55-kserve-5976ffff69-rm5rw\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:06.857971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:06.857949    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:07.293426 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:07.293401    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"]
Nov 01 22:41:07.294470 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:41:07.294441    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1acf4ec_d463_4380_be2b_2da98bf7d984.slice/crio-4e47b7cfd1c96b0ce327083e3757e2f483c0bd76bde0c42549ec8198c40e7c1b WatchSource:0}: Error finding container 4e47b7cfd1c96b0ce327083e3757e2f483c0bd76bde0c42549ec8198c40e7c1b: Status 404 returned error can't find the container with id 4e47b7cfd1c96b0ce327083e3757e2f483c0bd76bde0c42549ec8198c40e7c1b
Nov 01 22:41:07.648919 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:07.648830    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" event={"ID":"c1acf4ec-d463-4380-be2b-2da98bf7d984","Type":"ContainerStarted","Data":"064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73"}
Nov 01 22:41:07.648919 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:07.648879    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" event={"ID":"c1acf4ec-d463-4380-be2b-2da98bf7d984","Type":"ContainerStarted","Data":"4e47b7cfd1c96b0ce327083e3757e2f483c0bd76bde0c42549ec8198c40e7c1b"}
Nov 01 22:41:07.858445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:07.858418    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:11.660953 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.660902    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podStartSLOduration=5.660888002 podStartE2EDuration="5.660888002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:07.796474486 +0000 UTC m=+1049.385673546" watchObservedRunningTime="2026-11-01 22:41:11.660888002 +0000 UTC m=+1053.250087062"
Nov 01 22:41:11.662105 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.662085    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"]
Nov 01 22:41:11.666125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.666105    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.742396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.742357    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-home\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.742555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.742535    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-model-cache\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.742610 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.742570    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwflm\" (UniqueName: \"kubernetes.io/projected/05a78acc-77a3-455d-9dd7-2e4f827fa20d-kube-api-access-hwflm\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.742687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.742664    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tmp-dir\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.742748 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.742728    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-dshm\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.742848 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.742826    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tls-certs\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.764660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.764633    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"distinct-w0-c54de0-kserve-self-signed-certs\""
Nov 01 22:41:11.767559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.767540    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"]
Nov 01 22:41:11.844082 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.844005    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-model-cache\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.844326 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.844303    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwflm\" (UniqueName: \"kubernetes.io/projected/05a78acc-77a3-455d-9dd7-2e4f827fa20d-kube-api-access-hwflm\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.844468 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.844453    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tmp-dir\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.844588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.844575    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-dshm\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.844704 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.844679    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-model-cache\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.844806 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.844681    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tls-certs\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.844913 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.844890    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-home\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.845472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.845438    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-home\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.846460 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.845860    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tmp-dir\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.850301 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.850273    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tls-certs\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.879677 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.878152    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-dshm\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.899606 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.899572    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwflm\" (UniqueName: \"kubernetes.io/projected/05a78acc-77a3-455d-9dd7-2e4f827fa20d-kube-api-access-hwflm\") pod \"distinct-w0-c54de0-kserve-6f9458768d-9cwzh\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:11.979661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:11.979630    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:12.410173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:12.410141    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"]
Nov 01 22:41:12.410862 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:41:12.410834    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05a78acc_77a3_455d_9dd7_2e4f827fa20d.slice/crio-53ca418684bb9fc5a87df7e3b781b029bf215f7ec44ce415d2b3882d985e230e WatchSource:0}: Error finding container 53ca418684bb9fc5a87df7e3b781b029bf215f7ec44ce415d2b3882d985e230e: Status 404 returned error can't find the container with id 53ca418684bb9fc5a87df7e3b781b029bf215f7ec44ce415d2b3882d985e230e
Nov 01 22:41:12.681845 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:12.681814    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" event={"ID":"05a78acc-77a3-455d-9dd7-2e4f827fa20d","Type":"ContainerStarted","Data":"7260ba2a2f933ac509113e0b3dbd4d2e0855e28f806afb20c096edc3b62850b6"}
Nov 01 22:41:12.682219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:12.681855    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" event={"ID":"05a78acc-77a3-455d-9dd7-2e4f827fa20d","Type":"ContainerStarted","Data":"53ca418684bb9fc5a87df7e3b781b029bf215f7ec44ce415d2b3882d985e230e"}
Nov 01 22:41:12.792753 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:12.792700    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podStartSLOduration=1.79268637 podStartE2EDuration="1.79268637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:12.790158536 +0000 UTC m=+1054.379357597" watchObservedRunningTime="2026-11-01 22:41:12.79268637 +0000 UTC m=+1054.381885430"
Nov 01 22:41:12.980611 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:12.980526    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:16.858263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:16.858236    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:16.871732 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:16.871708    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:17.726410 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:17.726385    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:41:18.812792 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:41:18.812757    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:41:20.810528 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:20.810500    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:41:21.981452 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:21.981421    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:22.111155 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:22.111124    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:22.777567 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:22.777537    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:41:43.168929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.168898    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"]
Nov 01 22:41:43.173812 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.173790    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.273298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.273257    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"]
Nov 01 22:41:43.286667 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.286636    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.286812 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.286696    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.286864 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.286808    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.286906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.286872    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2m7\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-kube-api-access-7m2m7\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.286958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.286909    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.287035 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.287003    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.287140 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.287119    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.287188 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.287156    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.287188 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.287178    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.388782 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.388749    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.388932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.388795    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.388932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.388838    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.388932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.388903    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2m7\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-kube-api-access-7m2m7\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.388934    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.388976    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.389089    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.389129    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389352 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.389160    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389352 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.389204    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389352 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.389329    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389520 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.389355    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389587 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.389517    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.389627 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.389606    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.391176 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.391155    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.391497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.391482    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.463928 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.463863    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2m7\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-kube-api-access-7m2m7\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.464290 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.464271    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.487358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.487335    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:43.914150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.914092    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:41:43.914365 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.914340    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:41:43.914445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.914398    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:41:43.914972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:43.914947    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"]
Nov 01 22:41:44.905192 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:44.905160    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" event={"ID":"e2548cff-7724-4e35-a7a1-c97c40f7dc5f","Type":"ContainerStarted","Data":"530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6"}
Nov 01 22:41:44.905609 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:44.905210    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" event={"ID":"e2548cff-7724-4e35-a7a1-c97c40f7dc5f","Type":"ContainerStarted","Data":"6233a986c1daaa3a9f92fef6ea74e9aff255c2ba86f8ff8924e23e6b672a3e1f"}
Nov 01 22:41:45.062540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:45.062475    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podStartSLOduration=2.062460799 podStartE2EDuration="2.062460799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:45.061574769 +0000 UTC m=+1086.650773831" watchObservedRunningTime="2026-11-01 22:41:45.062460799 +0000 UTC m=+1086.651659859"
Nov 01 22:41:45.487817 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:45.487785    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:45.489169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:45.489137    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 22:41:45.489305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:45.489191    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 22:41:46.487818 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:46.487789    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 22:41:46.488259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:46.487847    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 22:41:47.488315 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:47.488280    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 22:41:47.488732 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:47.488331    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 22:41:48.488380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:48.488346    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 22:41:48.488767 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:48.488396    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 22:41:48.813074 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:41:48.812983    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:41:49.488018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:49.487984    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 22:41:49.488218 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:49.488036    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 22:41:50.488392 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:50.488357    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 22:41:50.488869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:50.488414    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 22:41:51.494177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:51.494144    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:51.495363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:51.495341    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:51.496505 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:51.496482    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:41:55.105194 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.105152    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"]
Nov 01 22:41:55.109268 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.109251    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163015    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163213 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163167    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163275 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163213    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163275 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163238    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4947b6-04b4-45a2-b98b-19d92bd16520-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163422 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163397    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163431    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163546 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163522    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-656xs\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-kube-api-access-656xs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163611 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163580    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.163727 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.163713    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.215956 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.215928    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"]
Nov 01 22:41:55.264761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.264727    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.264934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.264772    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.264934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.264814    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.264934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.264851    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.264934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.264872    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4947b6-04b4-45a2-b98b-19d92bd16520-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.264934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.264914    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.264934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.264928    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.265305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.264963    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-656xs\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-kube-api-access-656xs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.265305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.265001    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.265305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.265269    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.265305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.265266    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.265481 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.265369    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.265548 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.265524    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.265773 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.265752    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4947b6-04b4-45a2-b98b-19d92bd16520-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.267409 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.267381    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.267731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.267706    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.320716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.320688    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-656xs\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-kube-api-access-656xs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.325083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.325056    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.424250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.424224    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:55.840996 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.840926    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"]
Nov 01 22:41:55.873808 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:41:55.873782    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4947b6_04b4_45a2_b98b_19d92bd16520.slice/crio-6fa517c021e10624d52669f9e190ba98bb9b15ac437a5a648bed124b5adda629 WatchSource:0}: Error finding container 6fa517c021e10624d52669f9e190ba98bb9b15ac437a5a648bed124b5adda629: Status 404 returned error can't find the container with id 6fa517c021e10624d52669f9e190ba98bb9b15ac437a5a648bed124b5adda629
Nov 01 22:41:55.876185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.876154    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:41:55.876270 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.876222    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:41:55.876270 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.876252    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:41:55.998433 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.998398    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" event={"ID":"db4947b6-04b4-45a2-b98b-19d92bd16520","Type":"ContainerStarted","Data":"c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018"}
Nov 01 22:41:55.998591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:55.998457    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" event={"ID":"db4947b6-04b4-45a2-b98b-19d92bd16520","Type":"ContainerStarted","Data":"6fa517c021e10624d52669f9e190ba98bb9b15ac437a5a648bed124b5adda629"}
Nov 01 22:41:56.124429 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:56.124318    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podStartSLOduration=1.124298827 podStartE2EDuration="1.124298827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:56.109011626 +0000 UTC m=+1097.698210688" watchObservedRunningTime="2026-11-01 22:41:56.124298827 +0000 UTC m=+1097.713497887"
Nov 01 22:41:56.425371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:56.425335    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:41:56.426971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:56.426945    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:41:56.427100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:56.427000    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 22:41:57.425166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:57.425131    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:41:57.425651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:57.425178    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 22:41:58.425558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:58.425521    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:41:58.425948 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:58.425584    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 22:41:59.424757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:59.424720    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:41:59.424938 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:41:59.424782    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 22:42:00.424742 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.424700    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:42:00.425293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.424770    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 22:42:00.668580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.668546    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"]
Nov 01 22:42:00.672230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.672214    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.760084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.747781    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-tmp-dir\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.760084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.747879    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e389acf-e06f-4432-bf17-2d302059ef43-tls-certs\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.760084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.747927    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-home\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.760084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.747957    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-dshm\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.760084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.748005    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-model-cache\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.760084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.759218    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd7qx\" (UniqueName: \"kubernetes.io/projected/9e389acf-e06f-4432-bf17-2d302059ef43-kube-api-access-cd7qx\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.767529 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.767505    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"openshift-service-ca.crt\""
Nov 01 22:42:00.767680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.767528    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"kube-root-ca.crt\""
Nov 01 22:42:00.767967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.767937    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"default-dockercfg-8s5xm\""
Nov 01 22:42:00.768072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.767976    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"distinct-2-w5-d6101b-kserve-self-signed-certs\""
Nov 01 22:42:00.772640 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.772622    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"]
Nov 01 22:42:00.893067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.883010    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-tmp-dir\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.893067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.883637    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-tmp-dir\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.893067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.883966    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e389acf-e06f-4432-bf17-2d302059ef43-tls-certs\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.893067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.884690    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-home\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.893067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.884724    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-dshm\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.893067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.885426    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-home\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.893067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.885569    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-model-cache\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.893067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.885649    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd7qx\" (UniqueName: \"kubernetes.io/projected/9e389acf-e06f-4432-bf17-2d302059ef43-kube-api-access-cd7qx\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.907076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.899622    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-model-cache\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.907076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.900512    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e389acf-e06f-4432-bf17-2d302059ef43-tls-certs\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.919564 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.919534    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-dshm\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.958953 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.958878    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd7qx\" (UniqueName: \"kubernetes.io/projected/9e389acf-e06f-4432-bf17-2d302059ef43-kube-api-access-cd7qx\") pod \"distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:00.985111 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:00.985087    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:01.425359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:01.425332    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:42:01.425733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:01.425390    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 22:42:01.459234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:01.459175    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"]
Nov 01 22:42:01.461872 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:42:01.461845    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e389acf_e06f_4432_bf17_2d302059ef43.slice/crio-eec9f7fb1b5fff1e967e171c3fa6960df3646802a4f41ebbd71f7816510fc1c3 WatchSource:0}: Error finding container eec9f7fb1b5fff1e967e171c3fa6960df3646802a4f41ebbd71f7816510fc1c3: Status 404 returned error can't find the container with id eec9f7fb1b5fff1e967e171c3fa6960df3646802a4f41ebbd71f7816510fc1c3
Nov 01 22:42:02.099834 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:02.099792    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" event={"ID":"9e389acf-e06f-4432-bf17-2d302059ef43","Type":"ContainerStarted","Data":"81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24"}
Nov 01 22:42:02.099834 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:02.099838    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" event={"ID":"9e389acf-e06f-4432-bf17-2d302059ef43","Type":"ContainerStarted","Data":"eec9f7fb1b5fff1e967e171c3fa6960df3646802a4f41ebbd71f7816510fc1c3"}
Nov 01 22:42:02.424556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:02.424526    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:42:02.424730 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:02.424579    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 22:42:02.986123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:02.986096    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:03.429261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:03.429222    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:42:03.429640 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:03.429616    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:42:03.430251 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:03.430233    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:42:03.638363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:03.638316    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podStartSLOduration=3.638303623 podStartE2EDuration="3.638303623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:02.219255081 +0000 UTC m=+1103.808454143" watchObservedRunningTime="2026-11-01 22:42:03.638303623 +0000 UTC m=+1105.227502684"
Nov 01 22:42:04.967535 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:04.967499    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:42:10.986150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:10.986119    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:10.998374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:10.998351    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:11.165887 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:11.165857    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:42:18.813799 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:42:18.813723    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:42:33.993795 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:42:33.993752    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:42:48.813431 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:42:48.813394    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:43:06.242914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:06.242880    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:43:10.564138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.564108    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"]
Nov 01 22:43:10.568529 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.568509    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.669487 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.669460    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"]
Nov 01 22:43:10.681608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681482    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578cf15-26e8-4b66-98e1-eb7af275381c-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.681608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681538    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.681608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681601    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.681858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681665    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.681858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681736    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.681858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681791    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.681858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681829    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.682157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681860    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l756p\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-kube-api-access-l756p\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.682157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.681891    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.782918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.782890    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.782929    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l756p\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-kube-api-access-l756p\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.782962    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783139    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578cf15-26e8-4b66-98e1-eb7af275381c-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783187    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783261    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783330    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783366    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783371    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783547 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783408    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783547 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783443    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783570    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783696    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.783946 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.783919    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578cf15-26e8-4b66-98e1-eb7af275381c-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.785765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.785742    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.785860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.785810    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.838016 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.837959    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.838701 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.838678    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l756p\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-kube-api-access-l756p\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:10.882278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:10.882260    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:11.323873 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:11.323842    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"]
Nov 01 22:43:11.356701 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:43:11.356671    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7578cf15_26e8_4b66_98e1_eb7af275381c.slice/crio-95a2946c2b0a292b7296b4a3f89c677d92aa5917c155390a6dda6f0770b6214b WatchSource:0}: Error finding container 95a2946c2b0a292b7296b4a3f89c677d92aa5917c155390a6dda6f0770b6214b: Status 404 returned error can't find the container with id 95a2946c2b0a292b7296b4a3f89c677d92aa5917c155390a6dda6f0770b6214b
Nov 01 22:43:11.359119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:11.359087    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:11.359203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:11.359155    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:11.359203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:11.359189    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:11.606219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:11.606133    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" event={"ID":"7578cf15-26e8-4b66-98e1-eb7af275381c","Type":"ContainerStarted","Data":"cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d"}
Nov 01 22:43:11.606219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:11.606192    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" event={"ID":"7578cf15-26e8-4b66-98e1-eb7af275381c","Type":"ContainerStarted","Data":"95a2946c2b0a292b7296b4a3f89c677d92aa5917c155390a6dda6f0770b6214b"}
Nov 01 22:43:11.721897 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:11.721849    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podStartSLOduration=1.7218355079999998 podStartE2EDuration="1.721835508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:11.718018869 +0000 UTC m=+1173.307217930" watchObservedRunningTime="2026-11-01 22:43:11.721835508 +0000 UTC m=+1173.311034569"
Nov 01 22:43:11.883284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:11.883211    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:12.884783 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:12.884750    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 22:43:12.885219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:12.884800    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 22:43:13.882638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:13.882608    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 22:43:13.882829 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:13.882657    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 22:43:14.887628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:14.887591    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 22:43:14.888266 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:14.888233    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 22:43:15.883326 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:15.883293    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 22:43:15.883520 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:15.883354    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 22:43:16.883552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:16.883523    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 22:43:16.883945 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:16.883570    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 22:43:17.883216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:17.883182    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 22:43:17.883398 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:17.883236    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 22:43:18.817731 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:43:18.817693    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:43:18.974549 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:18.974517    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:18.975494 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:18.975470    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:18.977007 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:18.976978    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:43:21.887467 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.887437    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"]
Nov 01 22:43:21.891773 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.891756    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:21.966388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.966352    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74d4432-900c-4bc6-b703-bcae0e67c109-tls-certs\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:21.966558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.966419    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-tmp-dir\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:21.966648 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.966620    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-dshm\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:21.966743 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.966723    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/c74d4432-900c-4bc6-b703-bcae0e67c109-kube-api-access-ksvwh\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:21.966797 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.966786    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-home\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:21.966879 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.966865    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-model-cache\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:21.990983 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.990957    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"distinct-2-w2-c768f3-kserve-self-signed-certs\""
Nov 01 22:43:21.994506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:21.994485    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"]
Nov 01 22:43:22.073063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.069874    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-dshm\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.073063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.069968    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/c74d4432-900c-4bc6-b703-bcae0e67c109-kube-api-access-ksvwh\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.073063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.070057    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-home\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.073063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.070153    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-model-cache\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.073063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.070188    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74d4432-900c-4bc6-b703-bcae0e67c109-tls-certs\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.073063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.070246    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-tmp-dir\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.073063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.070822    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-tmp-dir\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.076902 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.076492    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-model-cache\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.077355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.077333    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-home\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.110130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.107774    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-dshm\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.110130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.109831    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74d4432-900c-4bc6-b703-bcae0e67c109-tls-certs\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.131779 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.131755    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/c74d4432-900c-4bc6-b703-bcae0e67c109-kube-api-access-ksvwh\") pod \"distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.204324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.204295    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:22.675375 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:43:22.675341    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc74d4432_900c_4bc6_b703_bcae0e67c109.slice/crio-00d2e29188fc5f6e76450e3fa68a36209d36616662822035e37003e84c1edf15 WatchSource:0}: Error finding container 00d2e29188fc5f6e76450e3fa68a36209d36616662822035e37003e84c1edf15: Status 404 returned error can't find the container with id 00d2e29188fc5f6e76450e3fa68a36209d36616662822035e37003e84c1edf15
Nov 01 22:43:22.675375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.675368    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"]
Nov 01 22:43:22.698261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:22.698239    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" event={"ID":"c74d4432-900c-4bc6-b703-bcae0e67c109","Type":"ContainerStarted","Data":"00d2e29188fc5f6e76450e3fa68a36209d36616662822035e37003e84c1edf15"}
Nov 01 22:43:23.705822 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:23.705789    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" event={"ID":"c74d4432-900c-4bc6-b703-bcae0e67c109","Type":"ContainerStarted","Data":"227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c"}
Nov 01 22:43:23.815233 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:23.815180    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podStartSLOduration=2.815166116 podStartE2EDuration="2.815166116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:23.813832029 +0000 UTC m=+1185.403031092" watchObservedRunningTime="2026-11-01 22:43:23.815166116 +0000 UTC m=+1185.404365176"
Nov 01 22:43:24.204528 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:24.204499    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:32.205215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:32.205182    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:32.217777 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:32.217750    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:32.776570 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:32.776546    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:43:38.666325 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:38.666293    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:43:38.669916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:38.669896    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:43:41.970107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:41.970074    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"]
Nov 01 22:43:41.974641 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:41.974623    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.067772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.067741    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-home\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.067772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.067775    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786ns\" (UniqueName: \"kubernetes.io/projected/85be5c4e-2135-40a8-85eb-43a9021b30a2-kube-api-access-786ns\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.067964 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.067892    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-dshm\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.067964 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.067945    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-model-cache\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.068151 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.067973    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-tmp-dir\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.068238 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.068217    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85be5c4e-2135-40a8-85eb-43a9021b30a2-tls-certs\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.070671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.070652    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"distinct-2-w1-5b8fd2-kserve-self-signed-certs\""
Nov 01 22:43:42.073926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.073903    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"]
Nov 01 22:43:42.174905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.174865    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-dshm\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.175662 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.175635    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-model-cache\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.175849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.175833    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-tmp-dir\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.176034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.176009    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-model-cache\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.176482 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.176091    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85be5c4e-2135-40a8-85eb-43a9021b30a2-tls-certs\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.176722 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.176705    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-home\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.177158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.177141    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786ns\" (UniqueName: \"kubernetes.io/projected/85be5c4e-2135-40a8-85eb-43a9021b30a2-kube-api-access-786ns\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.177670 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.176422    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-tmp-dir\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.177787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.177075    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-home\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.195465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.181847    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-dshm\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.196924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.196898    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85be5c4e-2135-40a8-85eb-43a9021b30a2-tls-certs\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.231875 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.231803    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-786ns\" (UniqueName: \"kubernetes.io/projected/85be5c4e-2135-40a8-85eb-43a9021b30a2-kube-api-access-786ns\") pod \"distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.287063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.287013    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:42.722738 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.722710    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"]
Nov 01 22:43:42.723051 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:43:42.723007    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85be5c4e_2135_40a8_85eb_43a9021b30a2.slice/crio-c4bf48a3d487fe9809383d722c1c0bf99ace7b34a2ce3c0b72c356f492869625 WatchSource:0}: Error finding container c4bf48a3d487fe9809383d722c1c0bf99ace7b34a2ce3c0b72c356f492869625: Status 404 returned error can't find the container with id c4bf48a3d487fe9809383d722c1c0bf99ace7b34a2ce3c0b72c356f492869625
Nov 01 22:43:42.864461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.864428    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" event={"ID":"85be5c4e-2135-40a8-85eb-43a9021b30a2","Type":"ContainerStarted","Data":"9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b"}
Nov 01 22:43:42.864599 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.864472    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" event={"ID":"85be5c4e-2135-40a8-85eb-43a9021b30a2","Type":"ContainerStarted","Data":"c4bf48a3d487fe9809383d722c1c0bf99ace7b34a2ce3c0b72c356f492869625"}
Nov 01 22:43:42.975888 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:42.975795    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podStartSLOduration=1.9757810230000001 podStartE2EDuration="1.975781023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:42.97327125 +0000 UTC m=+1204.562470311" watchObservedRunningTime="2026-11-01 22:43:42.975781023 +0000 UTC m=+1204.564980084"
Nov 01 22:43:43.288127 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:43.288033    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:44.349580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.349544    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"]
Nov 01 22:43:44.354034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.354016    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428499 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428467    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428655 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428504    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428655 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428609    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428655 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428627    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad4a38-83eb-498d-b205-3b42607b5060-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428714    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428729    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428836 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428782    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428836 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428798    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8289w\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-kube-api-access-8289w\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.428906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.428878    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.452742 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.452712    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"]
Nov 01 22:43:44.530088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530029    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530098    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530139    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530160    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad4a38-83eb-498d-b205-3b42607b5060-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530252    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530272    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530345    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530372    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8289w\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-kube-api-access-8289w\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530420    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530505    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530711 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530658    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530768 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530735    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.530893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.530867    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.531209 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.531189    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad4a38-83eb-498d-b205-3b42607b5060-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.532510 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.532490    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.532832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.532809    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.583718 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.583692    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.584005 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.583986    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8289w\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-kube-api-access-8289w\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:44.669246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:44.669222    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:45.094111 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:45.094082    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"]
Nov 01 22:43:45.094236 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:43:45.094151    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ad4a38_83eb_498d_b205_3b42607b5060.slice/crio-51d6dc394bcfb8f9333f605a14772edf4b1291ac2d0d8a77175efed5b41482a9 WatchSource:0}: Error finding container 51d6dc394bcfb8f9333f605a14772edf4b1291ac2d0d8a77175efed5b41482a9: Status 404 returned error can't find the container with id 51d6dc394bcfb8f9333f605a14772edf4b1291ac2d0d8a77175efed5b41482a9
Nov 01 22:43:45.096576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:45.096546    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:45.096671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:45.096609    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:45.096671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:45.096637    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:45.958464 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:45.958422    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" event={"ID":"e9ad4a38-83eb-498d-b205-3b42607b5060","Type":"ContainerStarted","Data":"3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3"}
Nov 01 22:43:45.959010 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:45.958520    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" event={"ID":"e9ad4a38-83eb-498d-b205-3b42607b5060","Type":"ContainerStarted","Data":"51d6dc394bcfb8f9333f605a14772edf4b1291ac2d0d8a77175efed5b41482a9"}
Nov 01 22:43:46.122654 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:46.122606    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" podStartSLOduration=2.122592206 podStartE2EDuration="2.122592206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:46.12027401 +0000 UTC m=+1207.709473071" watchObservedRunningTime="2026-11-01 22:43:46.122592206 +0000 UTC m=+1207.711791267"
Nov 01 22:43:46.670295 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:46.670262    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:46.672113 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:46.672090    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:43:46.672198 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:46.672138    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:43:47.669715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:47.669674    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:43:47.670125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:47.669733    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:43:48.669967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:48.669930    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:43:48.670384 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:48.669978    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:43:48.812367 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:43:48.812336    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:43:49.669695 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:49.669660    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:43:49.669890 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:49.669730    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:43:50.670489 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:50.670450    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:43:50.670874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:50.670504    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:43:51.669790 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:51.669755    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:43:51.669933 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:51.669814    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:43:52.288101 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:52.288069    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:52.302180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:52.302155    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:52.673642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:52.673605    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:52.673991 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:52.673974    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:52.674468 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:52.674453    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 22:43:53.014438 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:53.014369    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:43:56.123905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.123876    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"]
Nov 01 22:43:56.128340 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.128316    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.230661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.230629    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"]
Nov 01 22:43:56.302593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302557    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.302593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302595    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.302823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302620    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e90043e-2824-4385-9c8c-ae50bfa42a60-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.302823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302744    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.302823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302793    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2kxs\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-kube-api-access-r2kxs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.302823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302816    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.302957 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302830    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.302957 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302872    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.302957 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.302932    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.404603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404520    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2kxs\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-kube-api-access-r2kxs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.404603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404565    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.404603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404587    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.404833 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404610    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.404833 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404684    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.404922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404877    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.404922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404909    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.405099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404950    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e90043e-2824-4385-9c8c-ae50bfa42a60-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.405099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.404996    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.405099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.405006    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.405269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.405110    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.405269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.405126    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.405389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.405365    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.405666 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.405643    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e90043e-2824-4385-9c8c-ae50bfa42a60-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.407028 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.407005    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.407313 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.407297    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.458579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.458546    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.458805 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.458780    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2kxs\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-kube-api-access-r2kxs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:56.742928 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:56.742898    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:57.210336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:57.210309    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"]
Nov 01 22:43:57.227005 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:43:57.226977    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e90043e_2824_4385_9c8c_ae50bfa42a60.slice/crio-adc19c732c1d7c2c5a2ab5332a650c6d52a51309d1bc88ea248e749b5627c89a WatchSource:0}: Error finding container adc19c732c1d7c2c5a2ab5332a650c6d52a51309d1bc88ea248e749b5627c89a: Status 404 returned error can't find the container with id adc19c732c1d7c2c5a2ab5332a650c6d52a51309d1bc88ea248e749b5627c89a
Nov 01 22:43:57.229702 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:57.229667    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:57.229791 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:57.229749    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:57.229859 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:57.229797    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:43:58.061318 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:58.061269    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" event={"ID":"6e90043e-2824-4385-9c8c-ae50bfa42a60","Type":"ContainerStarted","Data":"cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66"}
Nov 01 22:43:58.061570 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:58.061367    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" event={"ID":"6e90043e-2824-4385-9c8c-ae50bfa42a60","Type":"ContainerStarted","Data":"adc19c732c1d7c2c5a2ab5332a650c6d52a51309d1bc88ea248e749b5627c89a"}
Nov 01 22:43:58.220549 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:58.220500    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podStartSLOduration=2.220487163 podStartE2EDuration="2.220487163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:58.21691248 +0000 UTC m=+1219.806111543" watchObservedRunningTime="2026-11-01 22:43:58.220487163 +0000 UTC m=+1219.809686235"
Nov 01 22:43:58.755069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:58.754379    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:43:58.755569 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:58.755533    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:43:58.755710 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:58.755603    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 22:43:59.220997 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.220961    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"]
Nov 01 22:43:59.225074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.225057    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.269761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.269727    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.269881 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.269865    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.269956 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.269890    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.270023 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.270005    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.270146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.270124    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830f4875-c6ea-44d1-8d67-c38918d477a4-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.270204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.270192    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.270256 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.270211    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.270256 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.270245    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.270330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.270279    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldpz\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-kube-api-access-xldpz\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.371776 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.371737    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.371926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.371861    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.371926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.371887    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.372014 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.371998    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.372122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372102    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830f4875-c6ea-44d1-8d67-c38918d477a4-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.372420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372398    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.372588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372572    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.372752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372726    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.373186 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372905    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldpz\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-kube-api-access-xldpz\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.373442 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372504    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.373442 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372504    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.373442 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372824    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830f4875-c6ea-44d1-8d67-c38918d477a4-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.373442 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.372587    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.373442 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.373086    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.374343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.374239    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.378278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.374837    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.463153 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.432978    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"]
Nov 01 22:43:59.596016 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.595923    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.596819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.596780    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldpz\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-kube-api-access-xldpz\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:43:59.754969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.754934    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:43:59.755297 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.755257    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 22:43:59.859443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:43:59.859352    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:44:00.277250 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:44:00.277204    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830f4875_c6ea_44d1_8d67_c38918d477a4.slice/crio-ebf5ae74d5228843bc5f02808bde876c79b5b5250131e24489f8f7fbee494048 WatchSource:0}: Error finding container ebf5ae74d5228843bc5f02808bde876c79b5b5250131e24489f8f7fbee494048: Status 404 returned error can't find the container with id ebf5ae74d5228843bc5f02808bde876c79b5b5250131e24489f8f7fbee494048
Nov 01 22:44:00.277624 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:00.277604    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"]
Nov 01 22:44:00.279924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:00.279892    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:00.280019 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:00.279963    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:00.280019 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:00.279992    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:00.744390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:00.744353    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:44:00.744669 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:00.744638    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 22:44:01.082670 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:01.082583    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" event={"ID":"830f4875-c6ea-44d1-8d67-c38918d477a4","Type":"ContainerStarted","Data":"2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f"}
Nov 01 22:44:01.082670 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:01.082643    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" event={"ID":"830f4875-c6ea-44d1-8d67-c38918d477a4","Type":"ContainerStarted","Data":"ebf5ae74d5228843bc5f02808bde876c79b5b5250131e24489f8f7fbee494048"}
Nov 01 22:44:01.288452 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:01.288372    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podStartSLOduration=2.288351495 podStartE2EDuration="2.288351495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:01.252534244 +0000 UTC m=+1222.841733308" watchObservedRunningTime="2026-11-01 22:44:01.288351495 +0000 UTC m=+1222.877550558"
Nov 01 22:44:01.747068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:01.743310    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:44:01.747068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:01.743400    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 22:44:01.869350 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:01.867474    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:44:01.876396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:01.874545    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:44:01.876396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:01.874602    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:44:02.747069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:02.743343    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:44:02.747069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:02.743405    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 22:44:02.874308 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:02.873441    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:44:02.874308 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:02.873500    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:44:03.746065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:03.745325    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:44:03.746065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:03.745396    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 22:44:03.861078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:03.860378    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:44:03.861078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:03.860442    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:44:04.880559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:04.880525    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:44:04.881210 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:04.881157    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:44:05.006263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:05.006214    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:44:05.007033 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:05.006757    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:44:05.008162 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:05.008138    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:44:05.860714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:05.860666    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:44:05.860903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:05.860732    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:44:06.859895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:06.859865    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:44:06.860414 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:06.859913    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:44:07.870527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:07.869722    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:44:07.870527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:07.870239    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:44:07.871501 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:07.871481    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:44:08.245785 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:08.245754    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:44:10.734383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:10.734332    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:44:10.758066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:10.757609    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:10.781064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:10.770254    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:10.815068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:10.784331    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:44:10.881181 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:10.881122    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.421180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.421133    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.568640 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.568566    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.847119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.821532    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:44:11.911060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.848138    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:44:11.911060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.908343    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:44:11.911060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.908417    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:44:11.911060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.848689    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.911469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.890794    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.967420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.848721    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 22:44:11.967420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.938135    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:44:11.967420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.938198    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:44:11.967420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.938230    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:44:11.967420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:11.848753    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:44:12.031068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.004921    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.063090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.053825    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.158204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.158116    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.424626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.424588    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.439506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.429515    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:44:12.485632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.485608    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:44:12.485889 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.485878    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:44:12.682135 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.664341    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:44:12.683614 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.683591    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:44:12.683833 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.683822    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:44:12.791234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:12.791206    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:44:16.009738 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.009706    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"]
Nov 01 22:44:16.015067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.015033    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.071021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.070999    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:44:16.110909 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.110889    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:44:16.116762 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.116737    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"]
Nov 01 22:44:16.141621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.141598    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.141707 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.141626    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.141844 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.141824    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdr8\" (UniqueName: \"kubernetes.io/projected/839b7f71-5fc5-450c-91f5-3bacd72e8eac-kube-api-access-qfdr8\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.242615 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.242581    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdr8\" (UniqueName: \"kubernetes.io/projected/839b7f71-5fc5-450c-91f5-3bacd72e8eac-kube-api-access-qfdr8\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.242769 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.242699    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.242769 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.242717    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.243417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.243394    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.245118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.245095    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.297987 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.297913    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdr8\" (UniqueName: \"kubernetes.io/projected/839b7f71-5fc5-450c-91f5-3bacd72e8eac-kube-api-access-qfdr8\") pod \"maas-api-revoke-sub-1ceef79c-lvdpz\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.329884 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.329861    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:16.706955 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:44:16.706915    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839b7f71_5fc5_450c_91f5_3bacd72e8eac.slice/crio-146366b6538f70734438450162e48bdfc631f6e6f2d24c00d1010d1e2b308b65 WatchSource:0}: Error finding container 146366b6538f70734438450162e48bdfc631f6e6f2d24c00d1010d1e2b308b65: Status 404 returned error can't find the container with id 146366b6538f70734438450162e48bdfc631f6e6f2d24c00d1010d1e2b308b65
Nov 01 22:44:16.707213 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:16.707194    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"]
Nov 01 22:44:17.487929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:17.487897    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz" event={"ID":"839b7f71-5fc5-450c-91f5-3bacd72e8eac","Type":"ContainerStarted","Data":"4826b3ddf750a4906f81dc7a58cab20634244d4fb39c6c590e9d13ef4291d401"}
Nov 01 22:44:17.488329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:17.487954    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz" event={"ID":"839b7f71-5fc5-450c-91f5-3bacd72e8eac","Type":"ContainerStarted","Data":"146366b6538f70734438450162e48bdfc631f6e6f2d24c00d1010d1e2b308b65"}
Nov 01 22:44:17.601050 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:17.601000    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz" podStartSLOduration=2.600986437 podStartE2EDuration="2.600986437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:17.599352478 +0000 UTC m=+1239.188551539" watchObservedRunningTime="2026-11-01 22:44:17.600986437 +0000 UTC m=+1239.190185495"
Nov 01 22:44:18.496211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:18.496181    4837 generic.go:356] "Generic (PLEG): container finished" podID="839b7f71-5fc5-450c-91f5-3bacd72e8eac" containerID="4826b3ddf750a4906f81dc7a58cab20634244d4fb39c6c590e9d13ef4291d401" exitCode=0
Nov 01 22:44:18.496603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:18.496265    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz" event={"ID":"839b7f71-5fc5-450c-91f5-3bacd72e8eac","Type":"ContainerDied","Data":"4826b3ddf750a4906f81dc7a58cab20634244d4fb39c6c590e9d13ef4291d401"}
Nov 01 22:44:18.812596 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:44:18.812509    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:44:19.676941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.676916    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:19.785276 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.785195    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-openshift-ca\") pod \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") "
Nov 01 22:44:19.785276 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.785254    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-xks-ca\") pod \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") "
Nov 01 22:44:19.785462 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.785280    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839b7f71-5fc5-450c-91f5-3bacd72e8eac-kube-api-access-qfdr8\" (UniqueName: \"kubernetes.io/projected/839b7f71-5fc5-450c-91f5-3bacd72e8eac-kube-api-access-qfdr8\") pod \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\" (UID: \"839b7f71-5fc5-450c-91f5-3bacd72e8eac\") "
Nov 01 22:44:19.785638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.785610    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-openshift-ca" pod "839b7f71-5fc5-450c-91f5-3bacd72e8eac" (UID: "839b7f71-5fc5-450c-91f5-3bacd72e8eac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:19.785960 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.785934    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:19.796187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.796152    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-xks-ca" pod "839b7f71-5fc5-450c-91f5-3bacd72e8eac" (UID: "839b7f71-5fc5-450c-91f5-3bacd72e8eac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:19.812784 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.812757    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839b7f71-5fc5-450c-91f5-3bacd72e8eac-kube-api-access-qfdr8" pod "839b7f71-5fc5-450c-91f5-3bacd72e8eac" (UID: "839b7f71-5fc5-450c-91f5-3bacd72e8eac"). InnerVolumeSpecName "kube-api-access-qfdr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:19.887119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.887085    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b7f71-5fc5-450c-91f5-3bacd72e8eac-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:19.887119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:19.887119    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdr8\" (UniqueName: \"kubernetes.io/projected/839b7f71-5fc5-450c-91f5-3bacd72e8eac-kube-api-access-qfdr8\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:20.510873 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:20.510846    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"
Nov 01 22:44:20.510873 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:20.510855    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz" event={"ID":"839b7f71-5fc5-450c-91f5-3bacd72e8eac","Type":"ContainerDied","Data":"146366b6538f70734438450162e48bdfc631f6e6f2d24c00d1010d1e2b308b65"}
Nov 01 22:44:20.511093 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:20.510894    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="146366b6538f70734438450162e48bdfc631f6e6f2d24c00d1010d1e2b308b65"
Nov 01 22:44:33.226418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.226379    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"]
Nov 01 22:44:33.227146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.227127    4837 cpu_manager.go:414] "containerMap: removing container" podUID="839b7f71-5fc5-450c-91f5-3bacd72e8eac" containerName="revoke-keys"
Nov 01 22:44:33.227146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.227146    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839b7f71-5fc5-450c-91f5-3bacd72e8eac" containerName="revoke-keys"
Nov 01 22:44:33.227270 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.227252    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="839b7f71-5fc5-450c-91f5-3bacd72e8eac" containerName="revoke-keys"
Nov 01 22:44:33.230982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.230964    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.264878 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.264824    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c624m\" (UniqueName: \"kubernetes.io/projected/b962a16e-cb7b-4ed6-99ab-999944397b9b-kube-api-access-c624m\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.264990 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.264956    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.265156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.265143    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.326540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.326502    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:44:33.326750 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.326558    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:44:33.332473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.332446    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"]
Nov 01 22:44:33.365974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.365948    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.366106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.366091    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c624m\" (UniqueName: \"kubernetes.io/projected/b962a16e-cb7b-4ed6-99ab-999944397b9b-kube-api-access-c624m\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.366160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.366144    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.366842 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.366822    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.368338 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.368313    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.420143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.420119    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c624m\" (UniqueName: \"kubernetes.io/projected/b962a16e-cb7b-4ed6-99ab-999944397b9b-kube-api-access-c624m\") pod \"maas-api-revoke-sub-b5b9ee65-b5jxz\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.544758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.544690    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:33.915996 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:33.915959    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"]
Nov 01 22:44:33.916304 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:44:33.916278    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb962a16e_cb7b_4ed6_99ab_999944397b9b.slice/crio-2fa13f39574c5d488510b10d9dbfd5f5d29d1ceb83701b221117fdc81bc36199 WatchSource:0}: Error finding container 2fa13f39574c5d488510b10d9dbfd5f5d29d1ceb83701b221117fdc81bc36199: Status 404 returned error can't find the container with id 2fa13f39574c5d488510b10d9dbfd5f5d29d1ceb83701b221117fdc81bc36199
Nov 01 22:44:34.047772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:34.047743    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"]
Nov 01 22:44:34.054971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:34.054945    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e04d8cc2-sczqt"]
Nov 01 22:44:34.526604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:34.526532    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f567f2-244a-4f74-9b23-4b4949b6ac10" path="/var/lib/kubelet/pods/66f567f2-244a-4f74-9b23-4b4949b6ac10/volumes"
Nov 01 22:44:34.601759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:34.601730    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz" event={"ID":"b962a16e-cb7b-4ed6-99ab-999944397b9b","Type":"ContainerStarted","Data":"5ddb629febe024b61ccf8ea1ea97996f50224e45d0a4b00ba3f3603f4347c534"}
Nov 01 22:44:34.601926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:34.601779    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz" event={"ID":"b962a16e-cb7b-4ed6-99ab-999944397b9b","Type":"ContainerStarted","Data":"2fa13f39574c5d488510b10d9dbfd5f5d29d1ceb83701b221117fdc81bc36199"}
Nov 01 22:44:34.721167 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:34.721100    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz" podStartSLOduration=1.721084645 podStartE2EDuration="1.721084645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:34.719200569 +0000 UTC m=+1256.308399661" watchObservedRunningTime="2026-11-01 22:44:34.721084645 +0000 UTC m=+1256.310283705"
Nov 01 22:44:35.610761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:35.610727    4837 generic.go:356] "Generic (PLEG): container finished" podID="b962a16e-cb7b-4ed6-99ab-999944397b9b" containerID="5ddb629febe024b61ccf8ea1ea97996f50224e45d0a4b00ba3f3603f4347c534" exitCode=0
Nov 01 22:44:35.611168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:35.610792    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz" event={"ID":"b962a16e-cb7b-4ed6-99ab-999944397b9b","Type":"ContainerDied","Data":"5ddb629febe024b61ccf8ea1ea97996f50224e45d0a4b00ba3f3603f4347c534"}
Nov 01 22:44:36.780624 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.780592    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:36.813415 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.813387    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-xks-ca\") pod \"b962a16e-cb7b-4ed6-99ab-999944397b9b\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") "
Nov 01 22:44:36.813553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.813437    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b962a16e-cb7b-4ed6-99ab-999944397b9b-kube-api-access-c624m\" (UniqueName: \"kubernetes.io/projected/b962a16e-cb7b-4ed6-99ab-999944397b9b-kube-api-access-c624m\") pod \"b962a16e-cb7b-4ed6-99ab-999944397b9b\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") "
Nov 01 22:44:36.813637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.813621    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-openshift-ca\") pod \"b962a16e-cb7b-4ed6-99ab-999944397b9b\" (UID: \"b962a16e-cb7b-4ed6-99ab-999944397b9b\") "
Nov 01 22:44:36.814076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.814031    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-openshift-ca" pod "b962a16e-cb7b-4ed6-99ab-999944397b9b" (UID: "b962a16e-cb7b-4ed6-99ab-999944397b9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:36.814366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.814336    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:36.815893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.815872    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b962a16e-cb7b-4ed6-99ab-999944397b9b-kube-api-access-c624m" pod "b962a16e-cb7b-4ed6-99ab-999944397b9b" (UID: "b962a16e-cb7b-4ed6-99ab-999944397b9b"). InnerVolumeSpecName "kube-api-access-c624m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:36.816001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.815928    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-xks-ca" pod "b962a16e-cb7b-4ed6-99ab-999944397b9b" (UID: "b962a16e-cb7b-4ed6-99ab-999944397b9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:36.915860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.915837    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b962a16e-cb7b-4ed6-99ab-999944397b9b-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:36.916023 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:36.915863    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c624m\" (UniqueName: \"kubernetes.io/projected/b962a16e-cb7b-4ed6-99ab-999944397b9b-kube-api-access-c624m\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:37.625122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.625087    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz" event={"ID":"b962a16e-cb7b-4ed6-99ab-999944397b9b","Type":"ContainerDied","Data":"2fa13f39574c5d488510b10d9dbfd5f5d29d1ceb83701b221117fdc81bc36199"}
Nov 01 22:44:37.625298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.625145    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fa13f39574c5d488510b10d9dbfd5f5d29d1ceb83701b221117fdc81bc36199"
Nov 01 22:44:37.625298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.625104    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"
Nov 01 22:44:37.770900 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.770870    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"]
Nov 01 22:44:37.771574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.771556    4837 cpu_manager.go:414] "containerMap: removing container" podUID="b962a16e-cb7b-4ed6-99ab-999944397b9b" containerName="revoke-keys"
Nov 01 22:44:37.771649 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.771575    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b962a16e-cb7b-4ed6-99ab-999944397b9b" containerName="revoke-keys"
Nov 01 22:44:37.771706 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.771678    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b962a16e-cb7b-4ed6-99ab-999944397b9b" containerName="revoke-keys"
Nov 01 22:44:37.775207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.775190    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.826024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.825993    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752lb\" (UniqueName: \"kubernetes.io/projected/7d1c7b45-8473-43da-860d-8057a1413da5-kube-api-access-752lb\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.826398 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.826145    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.826398 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.826381    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.826720 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.826697    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:44:37.827076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.827057    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:44:37.830775 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.830753    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"]
Nov 01 22:44:37.928057 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.928014    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.928201 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.928101    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752lb\" (UniqueName: \"kubernetes.io/projected/7d1c7b45-8473-43da-860d-8057a1413da5-kube-api-access-752lb\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.928201 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.928158    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.928771 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.928743    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.930776 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.930750    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:37.981990 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:37.981964    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-752lb\" (UniqueName: \"kubernetes.io/projected/7d1c7b45-8473-43da-860d-8057a1413da5-kube-api-access-752lb\") pod \"maas-api-revoke-sub-e91a3ead-lrxwt\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:38.088969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:38.088937    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:38.461769 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:38.461738    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"]
Nov 01 22:44:38.462798 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:44:38.462767    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1c7b45_8473_43da_860d_8057a1413da5.slice/crio-2abd74d20456d0f8533cd07358ef8b4868c7c933ef0a8a91f6388d51352adea8 WatchSource:0}: Error finding container 2abd74d20456d0f8533cd07358ef8b4868c7c933ef0a8a91f6388d51352adea8: Status 404 returned error can't find the container with id 2abd74d20456d0f8533cd07358ef8b4868c7c933ef0a8a91f6388d51352adea8
Nov 01 22:44:38.634561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:38.634529    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt" event={"ID":"7d1c7b45-8473-43da-860d-8057a1413da5","Type":"ContainerStarted","Data":"2abd74d20456d0f8533cd07358ef8b4868c7c933ef0a8a91f6388d51352adea8"}
Nov 01 22:44:39.643114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:39.643021    4837 generic.go:356] "Generic (PLEG): container finished" podID="7d1c7b45-8473-43da-860d-8057a1413da5" containerID="c9ba98aca4d3c3c087583984df55f9efe43735a1262a9a6b1bffa396cea6efdc" exitCode=0
Nov 01 22:44:39.643114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:39.643077    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt" event={"ID":"7d1c7b45-8473-43da-860d-8057a1413da5","Type":"ContainerDied","Data":"c9ba98aca4d3c3c087583984df55f9efe43735a1262a9a6b1bffa396cea6efdc"}
Nov 01 22:44:40.821122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.821099    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:40.872307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.872283    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1c7b45-8473-43da-860d-8057a1413da5-kube-api-access-752lb\" (UniqueName: \"kubernetes.io/projected/7d1c7b45-8473-43da-860d-8057a1413da5-kube-api-access-752lb\") pod \"7d1c7b45-8473-43da-860d-8057a1413da5\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") "
Nov 01 22:44:40.872426 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.872316    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-xks-ca\") pod \"7d1c7b45-8473-43da-860d-8057a1413da5\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") "
Nov 01 22:44:40.872426 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.872376    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-openshift-ca\") pod \"7d1c7b45-8473-43da-860d-8057a1413da5\" (UID: \"7d1c7b45-8473-43da-860d-8057a1413da5\") "
Nov 01 22:44:40.872815 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.872788    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-openshift-ca" pod "7d1c7b45-8473-43da-860d-8057a1413da5" (UID: "7d1c7b45-8473-43da-860d-8057a1413da5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:40.872952 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.872937    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:40.874378 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.874356    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1c7b45-8473-43da-860d-8057a1413da5-kube-api-access-752lb" pod "7d1c7b45-8473-43da-860d-8057a1413da5" (UID: "7d1c7b45-8473-43da-860d-8057a1413da5"). InnerVolumeSpecName "kube-api-access-752lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:40.874508 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.874488    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-xks-ca" pod "7d1c7b45-8473-43da-860d-8057a1413da5" (UID: "7d1c7b45-8473-43da-860d-8057a1413da5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:40.974017 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.973979    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-752lb\" (UniqueName: \"kubernetes.io/projected/7d1c7b45-8473-43da-860d-8057a1413da5-kube-api-access-752lb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:40.974017 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:40.974008    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1c7b45-8473-43da-860d-8057a1413da5-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:41.270606 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.270517    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"]
Nov 01 22:44:41.271148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.271130    4837 cpu_manager.go:414] "containerMap: removing container" podUID="7d1c7b45-8473-43da-860d-8057a1413da5" containerName="revoke-keys"
Nov 01 22:44:41.271253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.271148    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1c7b45-8473-43da-860d-8057a1413da5" containerName="revoke-keys"
Nov 01 22:44:41.271311 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.271285    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1c7b45-8473-43da-860d-8057a1413da5" containerName="revoke-keys"
Nov 01 22:44:41.274797 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.274782    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.328495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.328472    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"]
Nov 01 22:44:41.332421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.332401    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.375102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.375074    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"]
Nov 01 22:44:41.377735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.377700    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d92b6c6-a95e-4486-ab06-067af8b1d065-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.377846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.377743    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.377846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.377771    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv67p\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-kube-api-access-xv67p\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.377967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.377942    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.378018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.377989    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.378106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378086    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.378224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378186    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.378264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378251    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.378309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378280    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.378366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378323    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.378454 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378436    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.378502 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378469    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.378576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378560    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.378634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378590    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8nnc\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-kube-api-access-k8nnc\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.378738 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378724    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44200b05-aec9-4756-9bc8-d50ec570847a-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.378816 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378798    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44200b05-aec9-4756-9bc8-d50ec570847a-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.378877 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378859    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.378931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.378895    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.379887 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.379871    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"]
Nov 01 22:44:41.379971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.379957    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.383159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.383138    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"]
Nov 01 22:44:41.387018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.387003    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.471003 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.470193    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"]
Nov 01 22:44:41.476878 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.476845    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"]
Nov 01 22:44:41.480283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480219    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.480283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480248    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.480283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480285    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.480474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480451    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.480527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480486    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.480527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480514    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.480619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480551    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.480619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480593    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.480729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480619    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.480729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480657    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.480729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480668    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.480729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480701    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.480729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480727    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8nnc\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-kube-api-access-k8nnc\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.480998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480755    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzmx\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-kube-api-access-4bzmx\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.480998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480782    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.480998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480808    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.480998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480863    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.480998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480889    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.480998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480930    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44200b05-aec9-4756-9bc8-d50ec570847a-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.480998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.480931    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.481388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481197    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.481388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481209    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fnc\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-kube-api-access-p6fnc\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.481388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481280    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.481388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481334    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.481388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481369    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44200b05-aec9-4756-9bc8-d50ec570847a-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.481388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481391    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481413    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481438    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d92b6c6-a95e-4486-ab06-067af8b1d065-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481455    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481473    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481496    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481522    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv67p\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-kube-api-access-xv67p\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481581    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481604    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481624    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481650    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481666    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481685    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.481758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481701    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c7401d-383e-4522-8b28-68f5bf2413fc-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.482444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481868    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.482444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481894    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44200b05-aec9-4756-9bc8-d50ec570847a-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.482444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.481992    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.482444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.482355    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d92b6c6-a95e-4486-ab06-067af8b1d065-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.482444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.482388    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.482444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.482427    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.482750 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.482637    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.483924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.483904    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.495059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.485410    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44200b05-aec9-4756-9bc8-d50ec570847a-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.495059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.485694    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.495059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.488254    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.545815 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.545751    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"]
Nov 01 22:44:41.546762 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.546736    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.546918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.546899    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv67p\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-kube-api-access-xv67p\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.548303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.548282    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.548997 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.548966    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8nnc\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-kube-api-access-k8nnc\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.584293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584261    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.584444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584317    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.584444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584415    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.584583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584454    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.584583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584489    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.584583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584517    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c7401d-383e-4522-8b28-68f5bf2413fc-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.584583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584561    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.584810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584593    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.584810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584617    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.584810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584696    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.584810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584734    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.585015 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584828    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzmx\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-kube-api-access-4bzmx\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.585015 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584860    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.585015 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584868    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.585015 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584953    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.585265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.585015    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.585265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.585060    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.585265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.585107    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.585265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.584867    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.586137 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.586097    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.586498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.586471    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.586755 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.586734    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fnc\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-kube-api-access-p6fnc\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.586855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.586748    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c7401d-383e-4522-8b28-68f5bf2413fc-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.586855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.586788    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.587364 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.587235    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.587364 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.587357    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.587502 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.587486    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.587735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.587712    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.588574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.588552    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:41.590309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.590263    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.591019 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.591001    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.591197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.591007    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.591440 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.591419    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.642584 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.642532    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.642749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.642625    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.643870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.643845    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fnc\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-kube-api-access-p6fnc\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.644849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.644821    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:41.645266 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.645243    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzmx\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-kube-api-access-4bzmx\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:41.671076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.668571    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"
Nov 01 22:44:41.671076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.668585    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt" event={"ID":"7d1c7b45-8473-43da-860d-8057a1413da5","Type":"ContainerDied","Data":"2abd74d20456d0f8533cd07358ef8b4868c7c933ef0a8a91f6388d51352adea8"}
Nov 01 22:44:41.671076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.668631    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2abd74d20456d0f8533cd07358ef8b4868c7c933ef0a8a91f6388d51352adea8"
Nov 01 22:44:41.692921 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.692894    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:41.702366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:41.702334    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:42.051616 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.051527    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"]
Nov 01 22:44:42.154386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.148646    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.154386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.149151    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.154386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.149478    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.234457 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.234413    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.234574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.234544    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.234634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.234611    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.245245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.245215    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"]
Nov 01 22:44:42.248914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.248841    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.249172 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.248983    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.249172 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.249100    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.252134 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.251719    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"]
Nov 01 22:44:42.256783 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.256760    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"]
Nov 01 22:44:42.277979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.277952    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.278086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.278029    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.278146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.278105    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:42.833985 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.833941    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" event={"ID":"2d92b6c6-a95e-4486-ab06-067af8b1d065","Type":"ContainerStarted","Data":"f59eeb55b1c0ecd90a6c9648a4c07a6303bc12a8e3eca3d83b64cf2448d4dd35"}
Nov 01 22:44:42.834316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.834294    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" event={"ID":"2d92b6c6-a95e-4486-ab06-067af8b1d065","Type":"ContainerStarted","Data":"4ee2d54596090c35dc9524e8145e2d3c3749094c8bd249fd9cfb7e6bd8eba710"}
Nov 01 22:44:42.861811 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.860823    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" event={"ID":"44200b05-aec9-4756-9bc8-d50ec570847a","Type":"ContainerStarted","Data":"7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707"}
Nov 01 22:44:42.861811 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.860895    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" event={"ID":"44200b05-aec9-4756-9bc8-d50ec570847a","Type":"ContainerStarted","Data":"db4898f0b869a1011e35c163f270b5d5523f933497b3a43b68b3d917ea93684b"}
Nov 01 22:44:42.919063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.904435    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" event={"ID":"15c7401d-383e-4522-8b28-68f5bf2413fc","Type":"ContainerStarted","Data":"f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe"}
Nov 01 22:44:42.919063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.904502    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" event={"ID":"15c7401d-383e-4522-8b28-68f5bf2413fc","Type":"ContainerStarted","Data":"dc02bccd3f2aab6a5d2123ed5c9099b6f9bb07e50a6555a8df0495e0a990a138"}
Nov 01 22:44:42.957228 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.957193    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" event={"ID":"a3339144-9288-4e5d-b4f3-54b87e7dabb7","Type":"ContainerStarted","Data":"692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f"}
Nov 01 22:44:42.957500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:42.957476    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" event={"ID":"a3339144-9288-4e5d-b4f3-54b87e7dabb7","Type":"ContainerStarted","Data":"1e25a6a9c2c3f7726d1b354089002526507422109516bf9dbb4e22f18ab5861a"}
Nov 01 22:44:43.010066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.005712    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podStartSLOduration=2.005692655 podStartE2EDuration="2.005692655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:42.980891695 +0000 UTC m=+1264.570090757" watchObservedRunningTime="2026-11-01 22:44:43.005692655 +0000 UTC m=+1264.594891716"
Nov 01 22:44:43.144178 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.143776    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podStartSLOduration=2.143753933 podStartE2EDuration="2.143753933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:43.122385755 +0000 UTC m=+1264.711584816" watchObservedRunningTime="2026-11-01 22:44:43.143753933 +0000 UTC m=+1264.732952994"
Nov 01 22:44:43.328067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.327136    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podStartSLOduration=2.327102681 podStartE2EDuration="2.327102681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:43.326228158 +0000 UTC m=+1264.915427223" watchObservedRunningTime="2026-11-01 22:44:43.327102681 +0000 UTC m=+1264.916301743"
Nov 01 22:44:43.593368 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.593339    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:43.604164 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.604134    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.604360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.604330    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:43.729644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.729600    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:43.729937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.729907    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:43.730107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.730094    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:43.742066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.730377    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.742066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.730430    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:43.742066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.730519    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.742066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.730542    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:43.770577 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.770127    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:44:43.770924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.770902    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.771100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.771076    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:43.812227 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:43.812174    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.052264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.052169    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.488278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.488226    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:44:44.563337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.563289    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.570075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.565286    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.608149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.608104    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:44.608860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.608733    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.608860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.608783    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:44.647621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.647565    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:44:44.692067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.691645    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.692067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.691729    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:44.726080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.705145    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.726080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.705232    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:44.726080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.705145    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.726080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.705306    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:44.784982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.784868    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.813212 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.813165    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.828067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.820889    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:44:44.854284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.854258    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:44:45.015935 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.015877    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.045815 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.045691    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.096065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:44.989588    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podStartSLOduration=3.989566354 podStartE2EDuration="3.989566354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:43.472630784 +0000 UTC m=+1265.061829847" watchObservedRunningTime="2026-11-01 22:44:44.989566354 +0000 UTC m=+1266.578765417"
Nov 01 22:44:45.268937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.169171    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.268937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.196829    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.492218 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.492173    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.770123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.569979    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.771824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.571032    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.784135 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.571093    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.819285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.612241    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.819564 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.819529    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:45.819692 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.745276    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:44:45.819869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.819856    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:44:45.820000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.819989    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:44:45.821011 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.612296    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.825152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.747466    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.847242 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.847205    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:44:45.876822 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.847490    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.877061 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.876805    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:44:45.877959 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:45.793819    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.051118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.042145    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.051118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.042219    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:46.134541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.070535    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.134541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.072135    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.134541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.072179    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:46.134541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.072249    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.134541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.072271    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:46.134541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.090688    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:44:46.242106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.144092    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.242106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.144385    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:44:46.295007 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.294964    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.296032 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.296008    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:44:46.400905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.400800    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"]
Nov 01 22:44:46.685015 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.684980    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.814079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.813851    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:46.814079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.774371    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.814303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.741657    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:44:46.814303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.814245    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:44:46.833074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.697572    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.833074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.815154    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:46.833074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.714416    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.833074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.815201    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:46.833074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.751978    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.833074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.815239    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:46.833074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.774408    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.833074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.788146    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.874066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.870591    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.874066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.870914    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.912185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.912123    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.929076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:46.925795    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80624021-6qs7t"]
Nov 01 22:44:47.066789 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.066686    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.084319 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.073374    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.136277 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.136243    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"]
Nov 01 22:44:47.164065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.145519    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.184334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.184300    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bbd381-fnbsz"]
Nov 01 22:44:47.393241 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.393136    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:44:47.491603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.491559    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.609469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.609433    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.609725 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.609698    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:47.655555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.650222    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.660559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.658553    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:47.699157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.698996    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.699157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.699073    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:47.719078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.710152    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.719078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.710204    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:47.832913 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.832880    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:44:47.863066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.853973    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:44:47.863066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:47.854531    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.048581 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.035429    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.141153 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.045800    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.141438 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.047532    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.143074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.082557    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.167211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.082604    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.243211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.243159    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.245956 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.245922    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.395123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.361695    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.426064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.423560    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.846274 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:48.846206    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.006069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.003145    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:44:49.006069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.004406    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:44:49.028118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.028084    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.028353 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.028325    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:49.120772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.120682    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.128063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.121010    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:49.139226 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.125524    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.139445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.139412    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:49.139572 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.125735    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.139691 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.139651    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:49.143172 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.143138    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.194534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.144729    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.204064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.186312    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.245919 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.187092    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.269448 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.245489    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.269982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.251183    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.363211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.363164    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.425144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.412868    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c49865f-a9ae-4c21-99cc-2ebf21e618aa" path="/var/lib/kubelet/pods/3c49865f-a9ae-4c21-99cc-2ebf21e618aa/volumes"
Nov 01 22:44:49.425144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.412970    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.425144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.413758    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1966a5f-6cff-4108-bd27-01c55d399c07" path="/var/lib/kubelet/pods/e1966a5f-6cff-4108-bd27-01c55d399c07/volumes"
Nov 01 22:44:49.425144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.414341    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:44:49.430231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.430184    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.680204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.680114    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.743067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.680434    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.743067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.741139    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:49.743067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.727777    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.743067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.741193    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:49.773067 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:44:49.740812    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:44:49.773067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.741027    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:49.773067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.757097    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.834068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:49.808471    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:50.009125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.005464    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.022401 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.012822    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.022936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.022902    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.023379 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.023360    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:44:50.329380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.208814    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.329697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.225271    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.354895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.246112    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.383072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.278184    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.383072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.278218    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.383072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.337473    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.439069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.427147    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.467069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.440791    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.485640 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.451445    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.611132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.602361    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.633816 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.633777    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:50.656508 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.656473    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.658351 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.658321    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:50.736068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.735429    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.736068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.735487    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:50.736327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.736301    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.736397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:50.736342    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:51.029068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.024140    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.029068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.024532    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.029068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.024814    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.405115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.405006    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.407074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.406025    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.407074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.406494    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.407074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.406923    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.410447 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.410412    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.428709 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.428659    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.489200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.489151    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.560180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.560134    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.599262 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.599228    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.599538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.599511    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:51.759124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.755519    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.759124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.755579    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:51.759124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.756343    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.759124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.756385    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:51.759124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.756742    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.759124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.756777    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:52.001099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:51.996192    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.025472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.025356    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.046118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.026226    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.047123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.037213    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.341124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.333517    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:44:52.341124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.337390    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:44:52.416080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.414443    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.416080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.414853    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.416080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.415207    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.450497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.450122    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.492068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.491228    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.528297 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.521091    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:44:52.691496 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.691464    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.691798 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.691755    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:52.709976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.699153    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.712965 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.710388    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 22:44:52.726367 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.726346    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.727592 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.727553    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:52.728735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.728714    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.737719 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.737694    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 01 22:44:52.744405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.741718    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:44:52.744405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.741822    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:44:52.744405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.741858    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:44:52.744405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.741902    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:44:52.744405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.741932    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:44:52.782939 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.781436    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:44:52.820187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.820158    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:44:52.839217 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:52.839193    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:44:53.651529 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:53.651138    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.651529 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:53.651194    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 22:44:53.713307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:53.713266    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.713495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:53.713326    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 22:44:54.294839 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.294778    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:54.296658 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.296626    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:54.301023 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.301001    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:44:54.310455 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.310431    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:54.311663 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.311643    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:54.324392 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.324364    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:44:54.668562 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.668535    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:54.669646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.669624    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:54.670605 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.670577    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:44:54.710028 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.710003    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:54.711430 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.711406    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:54.712277 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:54.712253    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:44:56.467723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.467688    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"]
Nov 01 22:44:56.472279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.472258    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.537059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.537003    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"]
Nov 01 22:44:56.562618 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.562570    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.562787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.562718    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.562887 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.562785    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.562887 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.562844    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.562998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.562906    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.562998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.562973    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.563152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.563120    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.563221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.563199    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.563276 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.563226    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmzs\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-kube-api-access-wlmzs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664386    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664420    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmzs\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-kube-api-access-wlmzs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664656 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664473    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664656 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664517    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664656 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664556    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664656 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664575    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664656 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664605    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664685    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664772    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.664929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.664893    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.665121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.665091    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.665204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.665128    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.665204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.665195    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.665420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.665401    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.667018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.666997    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.667298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.667279    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.719749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.719679    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.721739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.721707    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmzs\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-kube-api-access-wlmzs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:56.789603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:56.789573    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:57.222515 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:57.222478    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"]
Nov 01 22:44:57.223017 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:44:57.222994    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c49f6c_74f6_4a86_ae15_b56eb5355230.slice/crio-140c9c689411399921f9ce074cc9cbbc6271691a9b5771f5dc040ff12a62b121 WatchSource:0}: Error finding container 140c9c689411399921f9ce074cc9cbbc6271691a9b5771f5dc040ff12a62b121: Status 404 returned error can't find the container with id 140c9c689411399921f9ce074cc9cbbc6271691a9b5771f5dc040ff12a62b121
Nov 01 22:44:57.225586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:57.225554    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:57.225650 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:57.225621    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:57.225686 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:57.225653    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:44:57.963479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:57.963442    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" event={"ID":"f7c49f6c-74f6-4a86-ae15-b56eb5355230","Type":"ContainerStarted","Data":"0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c"}
Nov 01 22:44:57.963876 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:57.963497    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" event={"ID":"f7c49f6c-74f6-4a86-ae15-b56eb5355230","Type":"ContainerStarted","Data":"140c9c689411399921f9ce074cc9cbbc6271691a9b5771f5dc040ff12a62b121"}
Nov 01 22:44:58.066970 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:58.066934    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"]
Nov 01 22:44:58.072091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:58.072063    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ab4a11-kdc62"]
Nov 01 22:44:58.110155 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:58.110111    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podStartSLOduration=2.110090139 podStartE2EDuration="2.110090139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:58.080797116 +0000 UTC m=+1279.669996180" watchObservedRunningTime="2026-11-01 22:44:58.110090139 +0000 UTC m=+1279.699289177"
Nov 01 22:44:58.530005 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:58.529974    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e799b3-e11d-4d64-ab5c-328e2abb44fd" path="/var/lib/kubelet/pods/f3e799b3-e11d-4d64-ab5c-328e2abb44fd/volumes"
Nov 01 22:44:58.790077 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:58.789980    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:44:58.791495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:58.791464    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 22:44:58.791650 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:58.791522    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 22:44:59.790628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:59.790591    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 22:44:59.791013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:44:59.790659    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 22:45:00.209211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.209181    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"]
Nov 01 22:45:00.214637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.214618    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:00.225838 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.225818    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"]
Nov 01 22:45:00.254064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.253221    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"]
Nov 01 22:45:00.256209 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.256166    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:00.258347 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.258323    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:00.274222 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.274194    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:45:00.276897 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.276874    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"]
Nov 01 22:45:00.281018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.281002    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:00.291654 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.291635    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"]
Nov 01 22:45:00.319158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.319131    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"]
Nov 01 22:45:00.364778 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.364745    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"]
Nov 01 22:45:00.368060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.366341    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed8b004-81c4-40af-b8b7-8687b787c2de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x\" (UID: \"eed8b004-81c4-40af-b8b7-8687b787c2de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:00.368060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.366423    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbnk\" (UniqueName: \"kubernetes.io/projected/eed8b004-81c4-40af-b8b7-8687b787c2de-kube-api-access-rqbnk\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x\" (UID: \"eed8b004-81c4-40af-b8b7-8687b787c2de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:00.368060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.366458    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4063873c-40dc-4bcb-9b5a-ef97d0728a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2\" (UID: \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:00.368060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.366491    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/4063873c-40dc-4bcb-9b5a-ef97d0728a53-kube-api-access-gsh7x\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2\" (UID: \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:00.368060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.366539    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg6lb\" (UniqueName: \"kubernetes.io/projected/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-kube-api-access-qg6lb\") pod \"maas-api-key-cleanup-29892885-2vkg5\" (UID: \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:00.368060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.366718    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89281266-4a7a-4c49-90a6-6bcd3ea1677c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk\" (UID: \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:00.368060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.366947    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-2vkg5\" (UID: \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:00.368060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.366981    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7nhm\" (UniqueName: \"kubernetes.io/projected/89281266-4a7a-4c49-90a6-6bcd3ea1677c-kube-api-access-b7nhm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk\" (UID: \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:00.393892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.393865    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"]
Nov 01 22:45:00.464119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.464020    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"]
Nov 01 22:45:00.470257 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.470222    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.485309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.485275    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89281266-4a7a-4c49-90a6-6bcd3ea1677c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk\" (UID: \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:00.485483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.485461    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-2vkg5\" (UID: \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:00.485537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.485502    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7nhm\" (UniqueName: \"kubernetes.io/projected/89281266-4a7a-4c49-90a6-6bcd3ea1677c-kube-api-access-b7nhm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk\" (UID: \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:00.485574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.485547    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed8b004-81c4-40af-b8b7-8687b787c2de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x\" (UID: \"eed8b004-81c4-40af-b8b7-8687b787c2de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:00.485615 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.485601    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbnk\" (UniqueName: \"kubernetes.io/projected/eed8b004-81c4-40af-b8b7-8687b787c2de-kube-api-access-rqbnk\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x\" (UID: \"eed8b004-81c4-40af-b8b7-8687b787c2de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:00.485669 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.485632    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4063873c-40dc-4bcb-9b5a-ef97d0728a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2\" (UID: \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:00.485723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.485666    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/4063873c-40dc-4bcb-9b5a-ef97d0728a53-kube-api-access-gsh7x\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2\" (UID: \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:00.485723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.485705    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6lb\" (UniqueName: \"kubernetes.io/projected/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-kube-api-access-qg6lb\") pod \"maas-api-key-cleanup-29892885-2vkg5\" (UID: \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:00.486079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.486038    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89281266-4a7a-4c49-90a6-6bcd3ea1677c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk\" (UID: \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:00.486379 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.486353    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed8b004-81c4-40af-b8b7-8687b787c2de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x\" (UID: \"eed8b004-81c4-40af-b8b7-8687b787c2de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:00.486458 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.486415    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-2vkg5\" (UID: \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:00.486458 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.486445    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4063873c-40dc-4bcb-9b5a-ef97d0728a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2\" (UID: \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:00.549907 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.549353    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:00.586941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.586903    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.599030 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.598749    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/4063873c-40dc-4bcb-9b5a-ef97d0728a53-kube-api-access-gsh7x\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2\" (UID: \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:00.599583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.599315    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz498\" (UniqueName: \"kubernetes.io/projected/323cb200-8abe-47de-820a-95d8c26e4fa2-kube-api-access-nz498\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.599804 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.599787    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.602169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.602148    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6lb\" (UniqueName: \"kubernetes.io/projected/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-kube-api-access-qg6lb\") pod \"maas-api-key-cleanup-29892885-2vkg5\" (UID: \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:00.603257 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.603241    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbnk\" (UniqueName: \"kubernetes.io/projected/eed8b004-81c4-40af-b8b7-8687b787c2de-kube-api-access-rqbnk\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x\" (UID: \"eed8b004-81c4-40af-b8b7-8687b787c2de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:00.605842 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.605819    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"]
Nov 01 22:45:00.607229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.607207    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7nhm\" (UniqueName: \"kubernetes.io/projected/89281266-4a7a-4c49-90a6-6bcd3ea1677c-kube-api-access-b7nhm\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk\" (UID: \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:00.701764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.701727    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.701918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.701895    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz498\" (UniqueName: \"kubernetes.io/projected/323cb200-8abe-47de-820a-95d8c26e4fa2-kube-api-access-nz498\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.701918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.701914    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.702377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.702358    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.704256 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.704211    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.757639 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.757557    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz498\" (UniqueName: \"kubernetes.io/projected/323cb200-8abe-47de-820a-95d8c26e4fa2-kube-api-access-nz498\") pod \"maas-api-revoke-sub-4ac09baa-wc4sr\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.790104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.789124    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:00.791329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.790911    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 22:45:00.791695 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.791456    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 22:45:00.829696 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.829670    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:00.881582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.881558    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:00.884070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.883350    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:00.913531 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:00.913440    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:01.214998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:01.214970    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"]
Nov 01 22:45:01.217814 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:01.217785    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323cb200_8abe_47de_820a_95d8c26e4fa2.slice/crio-cce3e92802588d8de5298add4ed1f3b025ab969479bbf46b2668ee7a564ddf70 WatchSource:0}: Error finding container cce3e92802588d8de5298add4ed1f3b025ab969479bbf46b2668ee7a564ddf70: Status 404 returned error can't find the container with id cce3e92802588d8de5298add4ed1f3b025ab969479bbf46b2668ee7a564ddf70
Nov 01 22:45:01.480287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:01.480144    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"]
Nov 01 22:45:01.568065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:01.558565    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"]
Nov 01 22:45:01.611212 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:01.611183    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"]
Nov 01 22:45:01.615328 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:01.615240    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4063873c_40dc_4bcb_9b5a_ef97d0728a53.slice/crio-aa2c7bcdd5e045145b1e17fe74e88cab9441269b1d7febad95d74b11b240ec7e WatchSource:0}: Error finding container aa2c7bcdd5e045145b1e17fe74e88cab9441269b1d7febad95d74b11b240ec7e: Status 404 returned error can't find the container with id aa2c7bcdd5e045145b1e17fe74e88cab9441269b1d7febad95d74b11b240ec7e
Nov 01 22:45:01.673634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:01.673604    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"]
Nov 01 22:45:01.683963 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:01.683941    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b077949_67e7_4b28_89b5_9fa2dd34e0d7.slice/crio-09ca5473b4d164a800c65aa84ed53b8fdb04607f3e23e81938944741d79bce05 WatchSource:0}: Error finding container 09ca5473b4d164a800c65aa84ed53b8fdb04607f3e23e81938944741d79bce05: Status 404 returned error can't find the container with id 09ca5473b4d164a800c65aa84ed53b8fdb04607f3e23e81938944741d79bce05
Nov 01 22:45:01.791060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:01.791015    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 22:45:01.791202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:01.791107    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 22:45:02.061936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.061898    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" event={"ID":"2b077949-67e7-4b28-89b5-9fa2dd34e0d7","Type":"ContainerStarted","Data":"09ca5473b4d164a800c65aa84ed53b8fdb04607f3e23e81938944741d79bce05"}
Nov 01 22:45:02.064147 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.064028    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" event={"ID":"4063873c-40dc-4bcb-9b5a-ef97d0728a53","Type":"ContainerStarted","Data":"aa2c7bcdd5e045145b1e17fe74e88cab9441269b1d7febad95d74b11b240ec7e"}
Nov 01 22:45:02.065767 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.065742    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" event={"ID":"eed8b004-81c4-40af-b8b7-8687b787c2de","Type":"ContainerStarted","Data":"312e89c5cb795b3deb9534d3129ef2ec1ee93c1f2accbcc7c9ac3443e0b3b671"}
Nov 01 22:45:02.068345 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.068320    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr" event={"ID":"323cb200-8abe-47de-820a-95d8c26e4fa2","Type":"ContainerStarted","Data":"49055a0cd78bc9728445222731e0bed6f0c3884b8cceebbf1410c547cce8c3af"}
Nov 01 22:45:02.069242 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.069218    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr" event={"ID":"323cb200-8abe-47de-820a-95d8c26e4fa2","Type":"ContainerStarted","Data":"cce3e92802588d8de5298add4ed1f3b025ab969479bbf46b2668ee7a564ddf70"}
Nov 01 22:45:02.072361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.072332    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" event={"ID":"89281266-4a7a-4c49-90a6-6bcd3ea1677c","Type":"ContainerStarted","Data":"98df895424c91f90520f82cb87e05377050a228ae0436c3020fdcfcfefc8ec8b"}
Nov 01 22:45:02.193261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.193219    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr" podStartSLOduration=2.193203605 podStartE2EDuration="2.193203605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:02.189078202 +0000 UTC m=+1283.778277262" watchObservedRunningTime="2026-11-01 22:45:02.193203605 +0000 UTC m=+1283.782402665"
Nov 01 22:45:02.790567 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.790539    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 22:45:02.790690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:02.790588    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 22:45:03.082136 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.082054    4837 generic.go:356] "Generic (PLEG): container finished" podID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerID="3957942a1253c3669dc4efc935298a2956f2f69f8e1591eb4c5ad93bab4b308d" exitCode=60
Nov 01 22:45:03.082537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.082135    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" event={"ID":"89281266-4a7a-4c49-90a6-6bcd3ea1677c","Type":"ContainerDied","Data":"3957942a1253c3669dc4efc935298a2956f2f69f8e1591eb4c5ad93bab4b308d"}
Nov 01 22:45:03.082537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.082423    4837 scope.go:122] "RemoveContainer" containerID="3957942a1253c3669dc4efc935298a2956f2f69f8e1591eb4c5ad93bab4b308d"
Nov 01 22:45:03.083876 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.083855    4837 generic.go:356] "Generic (PLEG): container finished" podID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerID="957684f19cd7c740530db5df9edaf67ab4aaf428739718c8ebc7e46e5837effc" exitCode=60
Nov 01 22:45:03.084001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.083881    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" event={"ID":"2b077949-67e7-4b28-89b5-9fa2dd34e0d7","Type":"ContainerDied","Data":"957684f19cd7c740530db5df9edaf67ab4aaf428739718c8ebc7e46e5837effc"}
Nov 01 22:45:03.084239 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.084224    4837 scope.go:122] "RemoveContainer" containerID="957684f19cd7c740530db5df9edaf67ab4aaf428739718c8ebc7e46e5837effc"
Nov 01 22:45:03.086319 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.086262    4837 generic.go:356] "Generic (PLEG): container finished" podID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerID="6086a6c7d73e58b11ef42341ccc78cd3b68f4c22adf8bc4cb1d88bfd9bc2a53a" exitCode=60
Nov 01 22:45:03.086535 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.086334    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" event={"ID":"4063873c-40dc-4bcb-9b5a-ef97d0728a53","Type":"ContainerDied","Data":"6086a6c7d73e58b11ef42341ccc78cd3b68f4c22adf8bc4cb1d88bfd9bc2a53a"}
Nov 01 22:45:03.086830 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.086659    4837 scope.go:122] "RemoveContainer" containerID="6086a6c7d73e58b11ef42341ccc78cd3b68f4c22adf8bc4cb1d88bfd9bc2a53a"
Nov 01 22:45:03.089072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.089035    4837 generic.go:356] "Generic (PLEG): container finished" podID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerID="41b9148f1f73de3dad2a585a22926f38133395c0cdd58b13e00fe5de6a002dab" exitCode=60
Nov 01 22:45:03.089191 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.089113    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" event={"ID":"eed8b004-81c4-40af-b8b7-8687b787c2de","Type":"ContainerDied","Data":"41b9148f1f73de3dad2a585a22926f38133395c0cdd58b13e00fe5de6a002dab"}
Nov 01 22:45:03.089494 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.089477    4837 scope.go:122] "RemoveContainer" containerID="41b9148f1f73de3dad2a585a22926f38133395c0cdd58b13e00fe5de6a002dab"
Nov 01 22:45:03.091487 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.091462    4837 generic.go:356] "Generic (PLEG): container finished" podID="323cb200-8abe-47de-820a-95d8c26e4fa2" containerID="49055a0cd78bc9728445222731e0bed6f0c3884b8cceebbf1410c547cce8c3af" exitCode=0
Nov 01 22:45:03.091565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.091534    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr" event={"ID":"323cb200-8abe-47de-820a-95d8c26e4fa2","Type":"ContainerDied","Data":"49055a0cd78bc9728445222731e0bed6f0c3884b8cceebbf1410c547cce8c3af"}
Nov 01 22:45:03.790148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.790113    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 01 22:45:03.790312 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:03.790162    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 01 22:45:04.102881 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.102849    4837 generic.go:356] "Generic (PLEG): container finished" podID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerID="743b4de23c0d28da926932d26e14752f4b180f09edb0fb237d5167ad0651c391" exitCode=60
Nov 01 22:45:04.103279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.102925    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" event={"ID":"89281266-4a7a-4c49-90a6-6bcd3ea1677c","Type":"ContainerDied","Data":"743b4de23c0d28da926932d26e14752f4b180f09edb0fb237d5167ad0651c391"}
Nov 01 22:45:04.103279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.102990    4837 scope.go:122] "RemoveContainer" containerID="3957942a1253c3669dc4efc935298a2956f2f69f8e1591eb4c5ad93bab4b308d"
Nov 01 22:45:04.103279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.103274    4837 scope.go:122] "RemoveContainer" containerID="743b4de23c0d28da926932d26e14752f4b180f09edb0fb237d5167ad0651c391"
Nov 01 22:45:04.103658 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:04.103637    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk_odh-ai-gateway-infra(89281266-4a7a-4c49-90a6-6bcd3ea1677c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c"
Nov 01 22:45:04.105452 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.105421    4837 generic.go:356] "Generic (PLEG): container finished" podID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerID="21ddef732e07b46e599f003b9bd453f16c46662017723513801cefe45ff7a8dd" exitCode=60
Nov 01 22:45:04.105541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.105497    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" event={"ID":"2b077949-67e7-4b28-89b5-9fa2dd34e0d7","Type":"ContainerDied","Data":"21ddef732e07b46e599f003b9bd453f16c46662017723513801cefe45ff7a8dd"}
Nov 01 22:45:04.105782 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.105762    4837 scope.go:122] "RemoveContainer" containerID="21ddef732e07b46e599f003b9bd453f16c46662017723513801cefe45ff7a8dd"
Nov 01 22:45:04.106133 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:04.106109    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-2vkg5_odh-ai-gateway-infra(2b077949-67e7-4b28-89b5-9fa2dd34e0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7"
Nov 01 22:45:04.107743 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.107719    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" event={"ID":"4063873c-40dc-4bcb-9b5a-ef97d0728a53","Type":"ContainerStarted","Data":"f09ead92b35fa8bf6cc8fda55504429fb2ab6cba7db25e24933b749d98e602ce"}
Nov 01 22:45:04.109885 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.109862    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" event={"ID":"eed8b004-81c4-40af-b8b7-8687b787c2de","Type":"ContainerStarted","Data":"020a4073dffad3664e3b9bff1d31784cff916aceb7e40e069b2e617a7466b67b"}
Nov 01 22:45:04.143387 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.143360    4837 scope.go:122] "RemoveContainer" containerID="957684f19cd7c740530db5df9edaf67ab4aaf428739718c8ebc7e46e5837effc"
Nov 01 22:45:04.347389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.347332    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" podStartSLOduration=4.347311016 podStartE2EDuration="4.347311016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.342293107 +0000 UTC m=+1285.931492168" watchObservedRunningTime="2026-11-01 22:45:04.347311016 +0000 UTC m=+1285.936510078"
Nov 01 22:45:04.391665 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.391644    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:04.540088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.540056    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323cb200-8abe-47de-820a-95d8c26e4fa2-kube-api-access-nz498\" (UniqueName: \"kubernetes.io/projected/323cb200-8abe-47de-820a-95d8c26e4fa2-kube-api-access-nz498\") pod \"323cb200-8abe-47de-820a-95d8c26e4fa2\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") "
Nov 01 22:45:04.540237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.540135    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-openshift-ca\") pod \"323cb200-8abe-47de-820a-95d8c26e4fa2\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") "
Nov 01 22:45:04.540237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.540224    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-xks-ca\") pod \"323cb200-8abe-47de-820a-95d8c26e4fa2\" (UID: \"323cb200-8abe-47de-820a-95d8c26e4fa2\") "
Nov 01 22:45:04.540498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.540467    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-openshift-ca" pod "323cb200-8abe-47de-820a-95d8c26e4fa2" (UID: "323cb200-8abe-47de-820a-95d8c26e4fa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:04.540943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.540921    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:04.542158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.542134    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-xks-ca" pod "323cb200-8abe-47de-820a-95d8c26e4fa2" (UID: "323cb200-8abe-47de-820a-95d8c26e4fa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:04.542249 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.542155    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323cb200-8abe-47de-820a-95d8c26e4fa2-kube-api-access-nz498" pod "323cb200-8abe-47de-820a-95d8c26e4fa2" (UID: "323cb200-8abe-47de-820a-95d8c26e4fa2"). InnerVolumeSpecName "kube-api-access-nz498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:04.575899 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.575846    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" podStartSLOduration=4.575829831 podStartE2EDuration="4.575829831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.573017207 +0000 UTC m=+1286.162216269" watchObservedRunningTime="2026-11-01 22:45:04.575829831 +0000 UTC m=+1286.165028893"
Nov 01 22:45:04.643269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.643202    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz498\" (UniqueName: \"kubernetes.io/projected/323cb200-8abe-47de-820a-95d8c26e4fa2-kube-api-access-nz498\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:04.643269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.643237    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323cb200-8abe-47de-820a-95d8c26e4fa2-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:04.793930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.793897    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:45:04.794326 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.794308    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:45:04.795071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:04.795054    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 22:45:05.081482 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.078621    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"]
Nov 01 22:45:05.093321 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.093289    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657793f-p2rsn"]
Nov 01 22:45:05.121943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.121921    4837 scope.go:122] "RemoveContainer" containerID="21ddef732e07b46e599f003b9bd453f16c46662017723513801cefe45ff7a8dd"
Nov 01 22:45:05.122372 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:05.122346    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-2vkg5_odh-ai-gateway-infra(2b077949-67e7-4b28-89b5-9fa2dd34e0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7"
Nov 01 22:45:05.123602 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.123577    4837 generic.go:356] "Generic (PLEG): container finished" podID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerID="f09ead92b35fa8bf6cc8fda55504429fb2ab6cba7db25e24933b749d98e602ce" exitCode=60
Nov 01 22:45:05.123719 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.123654    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" event={"ID":"4063873c-40dc-4bcb-9b5a-ef97d0728a53","Type":"ContainerDied","Data":"f09ead92b35fa8bf6cc8fda55504429fb2ab6cba7db25e24933b749d98e602ce"}
Nov 01 22:45:05.123719 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.123713    4837 scope.go:122] "RemoveContainer" containerID="6086a6c7d73e58b11ef42341ccc78cd3b68f4c22adf8bc4cb1d88bfd9bc2a53a"
Nov 01 22:45:05.123944 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.123923    4837 scope.go:122] "RemoveContainer" containerID="f09ead92b35fa8bf6cc8fda55504429fb2ab6cba7db25e24933b749d98e602ce"
Nov 01 22:45:05.124326 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:05.124302    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2_odh-ai-gateway-infra(4063873c-40dc-4bcb-9b5a-ef97d0728a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53"
Nov 01 22:45:05.125906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.125882    4837 generic.go:356] "Generic (PLEG): container finished" podID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerID="020a4073dffad3664e3b9bff1d31784cff916aceb7e40e069b2e617a7466b67b" exitCode=60
Nov 01 22:45:05.125972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.125955    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" event={"ID":"eed8b004-81c4-40af-b8b7-8687b787c2de","Type":"ContainerDied","Data":"020a4073dffad3664e3b9bff1d31784cff916aceb7e40e069b2e617a7466b67b"}
Nov 01 22:45:05.126203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.126189    4837 scope.go:122] "RemoveContainer" containerID="020a4073dffad3664e3b9bff1d31784cff916aceb7e40e069b2e617a7466b67b"
Nov 01 22:45:05.126500 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:05.126478    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x_odh-ai-gateway-infra(eed8b004-81c4-40af-b8b7-8687b787c2de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de"
Nov 01 22:45:05.127354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.127336    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"
Nov 01 22:45:05.127432 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.127361    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr" event={"ID":"323cb200-8abe-47de-820a-95d8c26e4fa2","Type":"ContainerDied","Data":"cce3e92802588d8de5298add4ed1f3b025ab969479bbf46b2668ee7a564ddf70"}
Nov 01 22:45:05.127432 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.127399    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cce3e92802588d8de5298add4ed1f3b025ab969479bbf46b2668ee7a564ddf70"
Nov 01 22:45:05.129435 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.129420    4837 scope.go:122] "RemoveContainer" containerID="743b4de23c0d28da926932d26e14752f4b180f09edb0fb237d5167ad0651c391"
Nov 01 22:45:05.129640 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:05.129623    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk_odh-ai-gateway-infra(89281266-4a7a-4c49-90a6-6bcd3ea1677c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c"
Nov 01 22:45:05.137976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:05.137958    4837 scope.go:122] "RemoveContainer" containerID="41b9148f1f73de3dad2a585a22926f38133395c0cdd58b13e00fe5de6a002dab"
Nov 01 22:45:06.138661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.138632    4837 scope.go:122] "RemoveContainer" containerID="f09ead92b35fa8bf6cc8fda55504429fb2ab6cba7db25e24933b749d98e602ce"
Nov 01 22:45:06.139210 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:06.138975    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2_odh-ai-gateway-infra(4063873c-40dc-4bcb-9b5a-ef97d0728a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53"
Nov 01 22:45:06.140348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.140330    4837 scope.go:122] "RemoveContainer" containerID="020a4073dffad3664e3b9bff1d31784cff916aceb7e40e069b2e617a7466b67b"
Nov 01 22:45:06.140559 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:06.140542    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x_odh-ai-gateway-infra(eed8b004-81c4-40af-b8b7-8687b787c2de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de"
Nov 01 22:45:06.243171 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.243143    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"]
Nov 01 22:45:06.243874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.243855    4837 cpu_manager.go:414] "containerMap: removing container" podUID="323cb200-8abe-47de-820a-95d8c26e4fa2" containerName="revoke-keys"
Nov 01 22:45:06.243874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.243874    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323cb200-8abe-47de-820a-95d8c26e4fa2" containerName="revoke-keys"
Nov 01 22:45:06.244002 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.243973    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="323cb200-8abe-47de-820a-95d8c26e4fa2" containerName="revoke-keys"
Nov 01 22:45:06.248884 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.248868    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.303795 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.303772    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"]
Nov 01 22:45:06.308202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.308181    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:06.367543 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.367520    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.367691 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.367671    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whfh\" (UniqueName: \"kubernetes.io/projected/908370d0-afcf-497c-b76f-889dc8ae01a3-kube-api-access-4whfh\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.367733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.367704    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.469169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.469144    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whfh\" (UniqueName: \"kubernetes.io/projected/908370d0-afcf-497c-b76f-889dc8ae01a3-kube-api-access-4whfh\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.469305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.469176    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.469344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.469310    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.469845 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.469824    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.471593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.471572    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.525419 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.525393    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whfh\" (UniqueName: \"kubernetes.io/projected/908370d0-afcf-497c-b76f-889dc8ae01a3-kube-api-access-4whfh\") pod \"maas-api-revoke-sub-791655fd-hxkjq\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:06.528924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.528899    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7ad130-4925-4b98-a80a-0215770e39a8" path="/var/lib/kubelet/pods/4c7ad130-4925-4b98-a80a-0215770e39a8/volumes"
Nov 01 22:45:06.564821 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:06.564799    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:07.009425 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:07.009392    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908370d0_afcf_497c_b76f_889dc8ae01a3.slice/crio-972903d891c0f4c253aa98a2b4b2524b2ca6f56e08bd0a77243293df83b9ed12 WatchSource:0}: Error finding container 972903d891c0f4c253aa98a2b4b2524b2ca6f56e08bd0a77243293df83b9ed12: Status 404 returned error can't find the container with id 972903d891c0f4c253aa98a2b4b2524b2ca6f56e08bd0a77243293df83b9ed12
Nov 01 22:45:07.010322 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:07.010296    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"]
Nov 01 22:45:07.150130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:07.150092    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq" event={"ID":"908370d0-afcf-497c-b76f-889dc8ae01a3","Type":"ContainerStarted","Data":"972903d891c0f4c253aa98a2b4b2524b2ca6f56e08bd0a77243293df83b9ed12"}
Nov 01 22:45:08.159340 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:08.159312    4837 generic.go:356] "Generic (PLEG): container finished" podID="908370d0-afcf-497c-b76f-889dc8ae01a3" containerID="ec9920632c659b64608b841cc31b75fd57c583af90934aacc882ec9fd7bd8e2e" exitCode=0
Nov 01 22:45:08.159722 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:08.159352    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq" event={"ID":"908370d0-afcf-497c-b76f-889dc8ae01a3","Type":"ContainerDied","Data":"ec9920632c659b64608b841cc31b75fd57c583af90934aacc882ec9fd7bd8e2e"}
Nov 01 22:45:08.925574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:08.925541    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"]
Nov 01 22:45:08.931436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:08.931415    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:08.984065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:08.984028    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"]
Nov 01 22:45:09.015051 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.015015    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.015187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.015166    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.015348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.015323    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzzm\" (UniqueName: \"kubernetes.io/projected/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-kube-api-access-8dzzm\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.116680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.116655    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.116794 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.116744    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzzm\" (UniqueName: \"kubernetes.io/projected/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-kube-api-access-8dzzm\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.116838 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.116824    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.117474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.117454    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.119018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.118990    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.171730 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.171709    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzzm\" (UniqueName: \"kubernetes.io/projected/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-kube-api-access-8dzzm\") pod \"maas-api-revoke-sub-9d52b510-r4nqs\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.244545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.244508    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:09.347567 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.347539    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:09.421056 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.421014    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-openshift-ca\") pod \"908370d0-afcf-497c-b76f-889dc8ae01a3\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") "
Nov 01 22:45:09.421284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.421267    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908370d0-afcf-497c-b76f-889dc8ae01a3-kube-api-access-4whfh\" (UniqueName: \"kubernetes.io/projected/908370d0-afcf-497c-b76f-889dc8ae01a3-kube-api-access-4whfh\") pod \"908370d0-afcf-497c-b76f-889dc8ae01a3\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") "
Nov 01 22:45:09.421374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.421326    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-xks-ca\") pod \"908370d0-afcf-497c-b76f-889dc8ae01a3\" (UID: \"908370d0-afcf-497c-b76f-889dc8ae01a3\") "
Nov 01 22:45:09.421417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.421390    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-openshift-ca" pod "908370d0-afcf-497c-b76f-889dc8ae01a3" (UID: "908370d0-afcf-497c-b76f-889dc8ae01a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:09.421973 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.421953    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:09.423778 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.423750    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908370d0-afcf-497c-b76f-889dc8ae01a3-kube-api-access-4whfh" pod "908370d0-afcf-497c-b76f-889dc8ae01a3" (UID: "908370d0-afcf-497c-b76f-889dc8ae01a3"). InnerVolumeSpecName "kube-api-access-4whfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:09.423887 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.423863    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-xks-ca" pod "908370d0-afcf-497c-b76f-889dc8ae01a3" (UID: "908370d0-afcf-497c-b76f-889dc8ae01a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:09.523671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.523602    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whfh\" (UniqueName: \"kubernetes.io/projected/908370d0-afcf-497c-b76f-889dc8ae01a3-kube-api-access-4whfh\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:09.523671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.523630    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908370d0-afcf-497c-b76f-889dc8ae01a3-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:09.666528 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:09.666496    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbdba72_c4af_4d9f_95fd_fb3c68ec2d05.slice/crio-8f6bea6209d4ceda83f00b3c89fb5582b30054467d5065ad49a69c23b0c049e7 WatchSource:0}: Error finding container 8f6bea6209d4ceda83f00b3c89fb5582b30054467d5065ad49a69c23b0c049e7: Status 404 returned error can't find the container with id 8f6bea6209d4ceda83f00b3c89fb5582b30054467d5065ad49a69c23b0c049e7
Nov 01 22:45:09.667664 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:09.667635    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"]
Nov 01 22:45:10.174696 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:10.174670    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"
Nov 01 22:45:10.174696 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:10.174689    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq" event={"ID":"908370d0-afcf-497c-b76f-889dc8ae01a3","Type":"ContainerDied","Data":"972903d891c0f4c253aa98a2b4b2524b2ca6f56e08bd0a77243293df83b9ed12"}
Nov 01 22:45:10.175184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:10.174735    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="972903d891c0f4c253aa98a2b4b2524b2ca6f56e08bd0a77243293df83b9ed12"
Nov 01 22:45:10.176427 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:10.176402    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs" event={"ID":"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05","Type":"ContainerStarted","Data":"b332cc141ff3266f86ea5786c1f9e3c41d15a1c335b48dc9d860be2174b8c4e8"}
Nov 01 22:45:10.176528 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:10.176448    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs" event={"ID":"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05","Type":"ContainerStarted","Data":"8f6bea6209d4ceda83f00b3c89fb5582b30054467d5065ad49a69c23b0c049e7"}
Nov 01 22:45:10.288102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:10.288027    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs" podStartSLOduration=2.288013766 podStartE2EDuration="2.288013766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:10.286224341 +0000 UTC m=+1291.875423402" watchObservedRunningTime="2026-11-01 22:45:10.288013766 +0000 UTC m=+1291.877212827"
Nov 01 22:45:11.186771 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.186740    4837 generic.go:356] "Generic (PLEG): container finished" podID="1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05" containerID="b332cc141ff3266f86ea5786c1f9e3c41d15a1c335b48dc9d860be2174b8c4e8" exitCode=0
Nov 01 22:45:11.187216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.186796    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs" event={"ID":"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05","Type":"ContainerDied","Data":"b332cc141ff3266f86ea5786c1f9e3c41d15a1c335b48dc9d860be2174b8c4e8"}
Nov 01 22:45:11.731835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.731804    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"]
Nov 01 22:45:11.732526 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.732510    4837 cpu_manager.go:414] "containerMap: removing container" podUID="908370d0-afcf-497c-b76f-889dc8ae01a3" containerName="revoke-keys"
Nov 01 22:45:11.732587 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.732527    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908370d0-afcf-497c-b76f-889dc8ae01a3" containerName="revoke-keys"
Nov 01 22:45:11.732658 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.732649    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="908370d0-afcf-497c-b76f-889dc8ae01a3" containerName="revoke-keys"
Nov 01 22:45:11.736179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.736162    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.836974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.836945    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"]
Nov 01 22:45:11.854950 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.854914    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.855394 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.855371    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.855608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.855592    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.855825 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.855794    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.855987 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.855956    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.856163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.856147    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.856528 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.856498    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.856685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.856654    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.856822 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.856807    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x45tt\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-kube-api-access-x45tt\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.957937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.957910    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.957954    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958018    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958267    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958346 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958299    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958343    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x45tt\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-kube-api-access-x45tt\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958371    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958480 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958397    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958480 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958413    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958575 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958560    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958592    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958616    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.958798 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958772    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.959013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.958991    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.960631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.960609    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:11.960698 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:11.960615    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:12.013068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.012980    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:12.013307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.013287    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x45tt\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-kube-api-access-x45tt\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:12.052190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.052170    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:12.340314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.340279    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:12.466034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.466006    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-kube-api-access-8dzzm\" (UniqueName: \"kubernetes.io/projected/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-kube-api-access-8dzzm\") pod \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") "
Nov 01 22:45:12.466034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.466076    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-openshift-ca\") pod \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") "
Nov 01 22:45:12.466034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.466124    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-xks-ca\") pod \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\" (UID: \"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05\") "
Nov 01 22:45:12.466034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.466837    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-openshift-ca" pod "1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05" (UID: "1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:12.466034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.467287    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:12.469173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.468995    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-xks-ca" pod "1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05" (UID: "1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:12.469377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.469355    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-kube-api-access-8dzzm" pod "1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05" (UID: "1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05"). InnerVolumeSpecName "kube-api-access-8dzzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:12.490081 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:12.490030    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa38a5f_c2bc_46fb_999a_cf6ab6f3a4d1.slice/crio-fc78abbad0966eb5d368e91111b50b559d7a0485d870cf88e1ca2ea81375eaa5 WatchSource:0}: Error finding container fc78abbad0966eb5d368e91111b50b559d7a0485d870cf88e1ca2ea81375eaa5: Status 404 returned error can't find the container with id fc78abbad0966eb5d368e91111b50b559d7a0485d870cf88e1ca2ea81375eaa5
Nov 01 22:45:12.490577 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.490553    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"]
Nov 01 22:45:12.492818 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.492787    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:12.492901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.492859    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:12.492901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.492889    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:12.568632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.568605    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzzm\" (UniqueName: \"kubernetes.io/projected/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-kube-api-access-8dzzm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:12.568632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:12.568633    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:13.066857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.066821    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"]
Nov 01 22:45:13.082303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.082272    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bb9326-rcvkf"]
Nov 01 22:45:13.162501 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.162466    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"]
Nov 01 22:45:13.163482 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.163465    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05" containerName="revoke-keys"
Nov 01 22:45:13.163537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.163485    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05" containerName="revoke-keys"
Nov 01 22:45:13.163700 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.163689    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05" containerName="revoke-keys"
Nov 01 22:45:13.167643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.167623    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.202149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.202125    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"
Nov 01 22:45:13.202269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.202164    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs" event={"ID":"1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05","Type":"ContainerDied","Data":"8f6bea6209d4ceda83f00b3c89fb5582b30054467d5065ad49a69c23b0c049e7"}
Nov 01 22:45:13.202269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.202210    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f6bea6209d4ceda83f00b3c89fb5582b30054467d5065ad49a69c23b0c049e7"
Nov 01 22:45:13.204124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.204096    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" event={"ID":"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1","Type":"ContainerStarted","Data":"0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6"}
Nov 01 22:45:13.204231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.204139    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" event={"ID":"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1","Type":"ContainerStarted","Data":"fc78abbad0966eb5d368e91111b50b559d7a0485d870cf88e1ca2ea81375eaa5"}
Nov 01 22:45:13.261748 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.261313    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"]
Nov 01 22:45:13.280324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.280291    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.280485 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.280449    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlc9l\" (UniqueName: \"kubernetes.io/projected/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-kube-api-access-mlc9l\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.280696 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.280669    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.381807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.381763    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlc9l\" (UniqueName: \"kubernetes.io/projected/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-kube-api-access-mlc9l\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.382327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.381886    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.382327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.382223    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.382697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.382673    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.384592 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.384570    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.402007 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.401956    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podStartSLOduration=2.401944268 podStartE2EDuration="2.401944268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:13.39779671 +0000 UTC m=+1294.986995782" watchObservedRunningTime="2026-11-01 22:45:13.401944268 +0000 UTC m=+1294.991143329"
Nov 01 22:45:13.458733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.458694    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlc9l\" (UniqueName: \"kubernetes.io/projected/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-kube-api-access-mlc9l\") pod \"maas-api-revoke-sub-afcc330a-9lkkb\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.482970 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.482943    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:13.881315 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:13.881281    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d980712_fd80_4bf0_98e3_1aa34de5b4f5.slice/crio-a3fb5bc75b22d3b2a76d051541c0b8a20680afa2f9654385026f75718c4e8548 WatchSource:0}: Error finding container a3fb5bc75b22d3b2a76d051541c0b8a20680afa2f9654385026f75718c4e8548: Status 404 returned error can't find the container with id a3fb5bc75b22d3b2a76d051541c0b8a20680afa2f9654385026f75718c4e8548
Nov 01 22:45:13.882798 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:13.882776    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"]
Nov 01 22:45:14.056070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:14.053327    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:14.057178 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:14.057146    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:45:14.057314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:14.057202    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 01 22:45:14.212397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:14.212360    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb" event={"ID":"3d980712-fd80-4bf0-98e3-1aa34de5b4f5","Type":"ContainerStarted","Data":"a3fb5bc75b22d3b2a76d051541c0b8a20680afa2f9654385026f75718c4e8548"}
Nov 01 22:45:14.526159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:14.526084    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18979efd-25a5-4f49-97f0-fef4604faae1" path="/var/lib/kubelet/pods/18979efd-25a5-4f49-97f0-fef4604faae1/volumes"
Nov 01 22:45:15.055413 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:15.055379    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:45:15.055564 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:15.055438    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 01 22:45:15.220986 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:15.220911    4837 generic.go:356] "Generic (PLEG): container finished" podID="3d980712-fd80-4bf0-98e3-1aa34de5b4f5" containerID="b99eaed10cbda3cee75c9b8b649d77c54b7d5883d0ac6c55f3feaa39940f6d54" exitCode=0
Nov 01 22:45:15.221161 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:15.220994    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb" event={"ID":"3d980712-fd80-4bf0-98e3-1aa34de5b4f5","Type":"ContainerDied","Data":"b99eaed10cbda3cee75c9b8b649d77c54b7d5883d0ac6c55f3feaa39940f6d54"}
Nov 01 22:45:16.055066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.054385    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:45:16.055066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.054444    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 01 22:45:16.440065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.432494    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:16.517724 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.517698    4837 scope.go:122] "RemoveContainer" containerID="020a4073dffad3664e3b9bff1d31784cff916aceb7e40e069b2e617a7466b67b"
Nov 01 22:45:16.517933 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.517915    4837 scope.go:122] "RemoveContainer" containerID="743b4de23c0d28da926932d26e14752f4b180f09edb0fb237d5167ad0651c391"
Nov 01 22:45:16.522667 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.522644    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-kube-api-access-mlc9l\" (UniqueName: \"kubernetes.io/projected/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-kube-api-access-mlc9l\") pod \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") "
Nov 01 22:45:16.522794 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.522778    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-openshift-ca\") pod \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") "
Nov 01 22:45:16.522843 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.522821    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-xks-ca\") pod \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\" (UID: \"3d980712-fd80-4bf0-98e3-1aa34de5b4f5\") "
Nov 01 22:45:16.523171 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.523145    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-openshift-ca" pod "3d980712-fd80-4bf0-98e3-1aa34de5b4f5" (UID: "3d980712-fd80-4bf0-98e3-1aa34de5b4f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:16.523729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.523704    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:16.526088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.525068    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-kube-api-access-mlc9l" pod "3d980712-fd80-4bf0-98e3-1aa34de5b4f5" (UID: "3d980712-fd80-4bf0-98e3-1aa34de5b4f5"). InnerVolumeSpecName "kube-api-access-mlc9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:16.526088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.525280    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-xks-ca" pod "3d980712-fd80-4bf0-98e3-1aa34de5b4f5" (UID: "3d980712-fd80-4bf0-98e3-1aa34de5b4f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:16.624917 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.624890    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlc9l\" (UniqueName: \"kubernetes.io/projected/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-kube-api-access-mlc9l\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:16.624917 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:16.624917    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d980712-fd80-4bf0-98e3-1aa34de5b4f5-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:17.055112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.053387    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:45:17.055112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.053447    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 01 22:45:17.261464 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.261424    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" event={"ID":"89281266-4a7a-4c49-90a6-6bcd3ea1677c","Type":"ContainerStarted","Data":"efd81c1e70e64f6839e49dc6ea8740fd3fa9ed1c782302351cbe5bc4240b3429"}
Nov 01 22:45:17.263395 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.263369    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" event={"ID":"eed8b004-81c4-40af-b8b7-8687b787c2de","Type":"ContainerStarted","Data":"e992a7e5c00cd1c9fc3fe9aef6aa57ceea7194ac9c028ee30d4335b5dc8001eb"}
Nov 01 22:45:17.264718 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.264696    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb" event={"ID":"3d980712-fd80-4bf0-98e3-1aa34de5b4f5","Type":"ContainerDied","Data":"a3fb5bc75b22d3b2a76d051541c0b8a20680afa2f9654385026f75718c4e8548"}
Nov 01 22:45:17.264808 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.264726    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"
Nov 01 22:45:17.264808 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.264741    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3fb5bc75b22d3b2a76d051541c0b8a20680afa2f9654385026f75718c4e8548"
Nov 01 22:45:17.378084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.378017    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" podStartSLOduration=17.378004888 podStartE2EDuration="17.378004888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:17.374583896 +0000 UTC m=+1298.963782959" watchObservedRunningTime="2026-11-01 22:45:17.378004888 +0000 UTC m=+1298.967203948"
Nov 01 22:45:17.518140 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:17.518027    4837 scope.go:122] "RemoveContainer" containerID="f09ead92b35fa8bf6cc8fda55504429fb2ab6cba7db25e24933b749d98e602ce"
Nov 01 22:45:18.055120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.054381    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:45:18.055120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.054436    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 01 22:45:18.274895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.274863    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" event={"ID":"4063873c-40dc-4bcb-9b5a-ef97d0728a53","Type":"ContainerStarted","Data":"096fb01d0b99fb69ce6844a23025335b0ffb14fb69535185b62f4c1a96f3a551"}
Nov 01 22:45:18.277157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.277130    4837 generic.go:356] "Generic (PLEG): container finished" podID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerID="e992a7e5c00cd1c9fc3fe9aef6aa57ceea7194ac9c028ee30d4335b5dc8001eb" exitCode=60
Nov 01 22:45:18.277294 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.277199    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" event={"ID":"eed8b004-81c4-40af-b8b7-8687b787c2de","Type":"ContainerDied","Data":"e992a7e5c00cd1c9fc3fe9aef6aa57ceea7194ac9c028ee30d4335b5dc8001eb"}
Nov 01 22:45:18.277294 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.277258    4837 scope.go:122] "RemoveContainer" containerID="020a4073dffad3664e3b9bff1d31784cff916aceb7e40e069b2e617a7466b67b"
Nov 01 22:45:18.277635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.277619    4837 scope.go:122] "RemoveContainer" containerID="e992a7e5c00cd1c9fc3fe9aef6aa57ceea7194ac9c028ee30d4335b5dc8001eb"
Nov 01 22:45:18.278014 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:18.277994    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x_odh-ai-gateway-infra(eed8b004-81c4-40af-b8b7-8687b787c2de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de"
Nov 01 22:45:18.279145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.279125    4837 generic.go:356] "Generic (PLEG): container finished" podID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerID="efd81c1e70e64f6839e49dc6ea8740fd3fa9ed1c782302351cbe5bc4240b3429" exitCode=60
Nov 01 22:45:18.279223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.279159    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" event={"ID":"89281266-4a7a-4c49-90a6-6bcd3ea1677c","Type":"ContainerDied","Data":"efd81c1e70e64f6839e49dc6ea8740fd3fa9ed1c782302351cbe5bc4240b3429"}
Nov 01 22:45:18.279495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.279481    4837 scope.go:122] "RemoveContainer" containerID="efd81c1e70e64f6839e49dc6ea8740fd3fa9ed1c782302351cbe5bc4240b3429"
Nov 01 22:45:18.279742 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:18.279725    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk_odh-ai-gateway-infra(89281266-4a7a-4c49-90a6-6bcd3ea1677c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c"
Nov 01 22:45:18.290118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.290103    4837 scope.go:122] "RemoveContainer" containerID="743b4de23c0d28da926932d26e14752f4b180f09edb0fb237d5167ad0651c391"
Nov 01 22:45:18.427825 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.427796    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"]
Nov 01 22:45:18.517974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.517949    4837 scope.go:122] "RemoveContainer" containerID="21ddef732e07b46e599f003b9bd453f16c46662017723513801cefe45ff7a8dd"
Nov 01 22:45:18.592092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:18.592069    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"]
Nov 01 22:45:18.814937 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:18.814845    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:45:19.053223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.053200    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:45:19.053331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.053249    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 01 22:45:19.291764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.291729    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" event={"ID":"2b077949-67e7-4b28-89b5-9fa2dd34e0d7","Type":"ContainerStarted","Data":"2bb45bdf28a4efc0eb09650952dea5a73b552cc3f1520163c3ef3ae5f2a9656b"}
Nov 01 22:45:19.294132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.294107    4837 generic.go:356] "Generic (PLEG): container finished" podID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerID="096fb01d0b99fb69ce6844a23025335b0ffb14fb69535185b62f4c1a96f3a551" exitCode=60
Nov 01 22:45:19.294599 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.294201    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" event={"ID":"4063873c-40dc-4bcb-9b5a-ef97d0728a53","Type":"ContainerDied","Data":"096fb01d0b99fb69ce6844a23025335b0ffb14fb69535185b62f4c1a96f3a551"}
Nov 01 22:45:19.294599 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.294271    4837 scope.go:122] "RemoveContainer" containerID="f09ead92b35fa8bf6cc8fda55504429fb2ab6cba7db25e24933b749d98e602ce"
Nov 01 22:45:19.294854 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.294628    4837 scope.go:122] "RemoveContainer" containerID="096fb01d0b99fb69ce6844a23025335b0ffb14fb69535185b62f4c1a96f3a551"
Nov 01 22:45:19.294983 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:19.294955    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2_odh-ai-gateway-infra(4063873c-40dc-4bcb-9b5a-ef97d0728a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53"
Nov 01 22:45:19.448618 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.448585    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"]
Nov 01 22:45:19.516566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.516536    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:19.581277 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.581256    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:19.605671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.605645    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed8b004-81c4-40af-b8b7-8687b787c2de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed8b004-81c4-40af-b8b7-8687b787c2de-maas-api-service-ca\") pod \"eed8b004-81c4-40af-b8b7-8687b787c2de\" (UID: \"eed8b004-81c4-40af-b8b7-8687b787c2de\") "
Nov 01 22:45:19.605794 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.605771    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed8b004-81c4-40af-b8b7-8687b787c2de-kube-api-access-rqbnk\" (UniqueName: \"kubernetes.io/projected/eed8b004-81c4-40af-b8b7-8687b787c2de-kube-api-access-rqbnk\") pod \"eed8b004-81c4-40af-b8b7-8687b787c2de\" (UID: \"eed8b004-81c4-40af-b8b7-8687b787c2de\") "
Nov 01 22:45:19.605923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.605906    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89281266-4a7a-4c49-90a6-6bcd3ea1677c-kube-api-access-b7nhm\" (UniqueName: \"kubernetes.io/projected/89281266-4a7a-4c49-90a6-6bcd3ea1677c-kube-api-access-b7nhm\") pod \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\" (UID: \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\") "
Nov 01 22:45:19.606053 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.606028    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89281266-4a7a-4c49-90a6-6bcd3ea1677c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89281266-4a7a-4c49-90a6-6bcd3ea1677c-maas-api-service-ca\") pod \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\" (UID: \"89281266-4a7a-4c49-90a6-6bcd3ea1677c\") "
Nov 01 22:45:19.606183 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.606038    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed8b004-81c4-40af-b8b7-8687b787c2de-maas-api-service-ca" pod "eed8b004-81c4-40af-b8b7-8687b787c2de" (UID: "eed8b004-81c4-40af-b8b7-8687b787c2de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:19.606377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.606352    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89281266-4a7a-4c49-90a6-6bcd3ea1677c-maas-api-service-ca" pod "89281266-4a7a-4c49-90a6-6bcd3ea1677c" (UID: "89281266-4a7a-4c49-90a6-6bcd3ea1677c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:19.606672 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.606654    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed8b004-81c4-40af-b8b7-8687b787c2de-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:19.606754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.606677    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89281266-4a7a-4c49-90a6-6bcd3ea1677c-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:19.607914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.607896    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed8b004-81c4-40af-b8b7-8687b787c2de-kube-api-access-rqbnk" pod "eed8b004-81c4-40af-b8b7-8687b787c2de" (UID: "eed8b004-81c4-40af-b8b7-8687b787c2de"). InnerVolumeSpecName "kube-api-access-rqbnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:19.608106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.608088    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89281266-4a7a-4c49-90a6-6bcd3ea1677c-kube-api-access-b7nhm" pod "89281266-4a7a-4c49-90a6-6bcd3ea1677c" (UID: "89281266-4a7a-4c49-90a6-6bcd3ea1677c"). InnerVolumeSpecName "kube-api-access-b7nhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:19.708125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.708084    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqbnk\" (UniqueName: \"kubernetes.io/projected/eed8b004-81c4-40af-b8b7-8687b787c2de-kube-api-access-rqbnk\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:19.708125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.708119    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7nhm\" (UniqueName: \"kubernetes.io/projected/89281266-4a7a-4c49-90a6-6bcd3ea1677c-kube-api-access-b7nhm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:19.782542 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:19.782488    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" podStartSLOduration=19.782474817 podStartE2EDuration="19.782474817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:19.779502543 +0000 UTC m=+1301.368701606" watchObservedRunningTime="2026-11-01 22:45:19.782474817 +0000 UTC m=+1301.371673878"
Nov 01 22:45:20.056958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.056918    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:20.057445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.057421    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:20.058345 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.058321    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:45:20.340894 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.340806    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"
Nov 01 22:45:20.341583 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.340813    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk" event={"ID":"89281266-4a7a-4c49-90a6-6bcd3ea1677c","Type":"ContainerDied","Data":"98df895424c91f90520f82cb87e05377050a228ae0436c3020fdcfcfefc8ec8b"}
Nov 01 22:45:20.341682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.341626    4837 scope.go:122] "RemoveContainer" containerID="efd81c1e70e64f6839e49dc6ea8740fd3fa9ed1c782302351cbe5bc4240b3429"
Nov 01 22:45:20.344393 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.344367    4837 generic.go:356] "Generic (PLEG): container finished" podID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerID="2bb45bdf28a4efc0eb09650952dea5a73b552cc3f1520163c3ef3ae5f2a9656b" exitCode=60
Nov 01 22:45:20.344634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.344613    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" event={"ID":"2b077949-67e7-4b28-89b5-9fa2dd34e0d7","Type":"ContainerDied","Data":"2bb45bdf28a4efc0eb09650952dea5a73b552cc3f1520163c3ef3ae5f2a9656b"}
Nov 01 22:45:20.345540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.345517    4837 scope.go:122] "RemoveContainer" containerID="2bb45bdf28a4efc0eb09650952dea5a73b552cc3f1520163c3ef3ae5f2a9656b"
Nov 01 22:45:20.346035 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:20.346010    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-2vkg5_odh-ai-gateway-infra(2b077949-67e7-4b28-89b5-9fa2dd34e0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7"
Nov 01 22:45:20.354591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.354572    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"
Nov 01 22:45:20.359224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.359201    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x" event={"ID":"eed8b004-81c4-40af-b8b7-8687b787c2de","Type":"ContainerDied","Data":"312e89c5cb795b3deb9534d3129ef2ec1ee93c1f2accbcc7c9ac3443e0b3b671"}
Nov 01 22:45:20.362411 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.362392    4837 scope.go:122] "RemoveContainer" containerID="21ddef732e07b46e599f003b9bd453f16c46662017723513801cefe45ff7a8dd"
Nov 01 22:45:20.401088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.393331    4837 scope.go:122] "RemoveContainer" containerID="e992a7e5c00cd1c9fc3fe9aef6aa57ceea7194ac9c028ee30d4335b5dc8001eb"
Nov 01 22:45:20.530222 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.530182    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"]
Nov 01 22:45:20.530374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.530231    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892885-htv5x"]
Nov 01 22:45:20.568370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.568348    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:20.629026 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.627244    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4063873c-40dc-4bcb-9b5a-ef97d0728a53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4063873c-40dc-4bcb-9b5a-ef97d0728a53-maas-api-service-ca\") pod \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\" (UID: \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\") "
Nov 01 22:45:20.629026 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.627750    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4063873c-40dc-4bcb-9b5a-ef97d0728a53-maas-api-service-ca" pod "4063873c-40dc-4bcb-9b5a-ef97d0728a53" (UID: "4063873c-40dc-4bcb-9b5a-ef97d0728a53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:20.629026 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.628343    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4063873c-40dc-4bcb-9b5a-ef97d0728a53-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:20.729538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.729514    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4063873c-40dc-4bcb-9b5a-ef97d0728a53-kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/4063873c-40dc-4bcb-9b5a-ef97d0728a53-kube-api-access-gsh7x\") pod \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\" (UID: \"4063873c-40dc-4bcb-9b5a-ef97d0728a53\") "
Nov 01 22:45:20.731579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.731553    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4063873c-40dc-4bcb-9b5a-ef97d0728a53-kube-api-access-gsh7x" pod "4063873c-40dc-4bcb-9b5a-ef97d0728a53" (UID: "4063873c-40dc-4bcb-9b5a-ef97d0728a53"). InnerVolumeSpecName "kube-api-access-gsh7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:20.786438 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.786410    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"]
Nov 01 22:45:20.804316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.804283    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"]
Nov 01 22:45:20.808117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.808094    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892885-zkgmk"]
Nov 01 22:45:20.831569 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:20.831546    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsh7x\" (UniqueName: \"kubernetes.io/projected/4063873c-40dc-4bcb-9b5a-ef97d0728a53-kube-api-access-gsh7x\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:21.366372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.366334    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2" event={"ID":"4063873c-40dc-4bcb-9b5a-ef97d0728a53","Type":"ContainerDied","Data":"aa2c7bcdd5e045145b1e17fe74e88cab9441269b1d7febad95d74b11b240ec7e"}
Nov 01 22:45:21.366372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.366360    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"
Nov 01 22:45:21.366792 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.366412    4837 scope.go:122] "RemoveContainer" containerID="096fb01d0b99fb69ce6844a23025335b0ffb14fb69535185b62f4c1a96f3a551"
Nov 01 22:45:21.539623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.539594    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:21.587298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.587273    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"]
Nov 01 22:45:21.591674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.591652    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892885-f6ng2"]
Nov 01 22:45:21.641931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.641864    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-maas-api-service-ca\") pod \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\" (UID: \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\") "
Nov 01 22:45:21.642083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.641962    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-kube-api-access-qg6lb\" (UniqueName: \"kubernetes.io/projected/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-kube-api-access-qg6lb\") pod \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\" (UID: \"2b077949-67e7-4b28-89b5-9fa2dd34e0d7\") "
Nov 01 22:45:21.642263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.642234    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-maas-api-service-ca" pod "2b077949-67e7-4b28-89b5-9fa2dd34e0d7" (UID: "2b077949-67e7-4b28-89b5-9fa2dd34e0d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:21.642572 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.642554    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:21.643889 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.643868    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-kube-api-access-qg6lb" pod "2b077949-67e7-4b28-89b5-9fa2dd34e0d7" (UID: "2b077949-67e7-4b28-89b5-9fa2dd34e0d7"). InnerVolumeSpecName "kube-api-access-qg6lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:21.743731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:21.743686    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg6lb\" (UniqueName: \"kubernetes.io/projected/2b077949-67e7-4b28-89b5-9fa2dd34e0d7-kube-api-access-qg6lb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:22.376334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.376301    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"
Nov 01 22:45:22.376744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.376303    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5" event={"ID":"2b077949-67e7-4b28-89b5-9fa2dd34e0d7","Type":"ContainerDied","Data":"09ca5473b4d164a800c65aa84ed53b8fdb04607f3e23e81938944741d79bce05"}
Nov 01 22:45:22.376744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.376439    4837 scope.go:122] "RemoveContainer" containerID="2bb45bdf28a4efc0eb09650952dea5a73b552cc3f1520163c3ef3ae5f2a9656b"
Nov 01 22:45:22.527913 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.527884    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" path="/var/lib/kubelet/pods/4063873c-40dc-4bcb-9b5a-ef97d0728a53/volumes"
Nov 01 22:45:22.528509 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.528491    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" path="/var/lib/kubelet/pods/89281266-4a7a-4c49-90a6-6bcd3ea1677c/volumes"
Nov 01 22:45:22.529106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.529082    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" path="/var/lib/kubelet/pods/eed8b004-81c4-40af-b8b7-8687b787c2de/volumes"
Nov 01 22:45:22.532710 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.532686    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"]
Nov 01 22:45:22.538281 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.538262    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-2vkg5"]
Nov 01 22:45:22.601872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.601845    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"]
Nov 01 22:45:22.602524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602509    4837 cpu_manager.go:414] "containerMap: removing container" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602525    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602537    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3d980712-fd80-4bf0-98e3-1aa34de5b4f5" containerName="revoke-keys"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602542    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d980712-fd80-4bf0-98e3-1aa34de5b4f5" containerName="revoke-keys"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602552    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602558    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602563    4837 cpu_manager.go:414] "containerMap: removing container" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602568    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602578    4837 cpu_manager.go:414] "containerMap: removing container" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:22.602582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602583    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602589    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602594    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602616    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602620    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602636    4837 cpu_manager.go:414] "containerMap: removing container" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602641    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602648    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602652    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602661    4837 cpu_manager.go:414] "containerMap: removing container" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602665    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602671    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602675    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602780    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602789    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602795    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602804    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602809    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602820    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602827    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d980712-fd80-4bf0-98e3-1aa34de5b4f5" containerName="revoke-keys"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602837    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:22.602858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.602841    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:22.607787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.607768    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.659492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.659439    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:22.659936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.659581    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:45:22.660442 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.660419    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sknkf\" (UniqueName: \"kubernetes.io/projected/f48c3035-122d-4c47-9310-fc188aab058b-kube-api-access-sknkf\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.660633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.660606    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.660736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.660673    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.664628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.664608    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"]
Nov 01 22:45:22.727388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.727362    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:45:22.762012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.761986    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sknkf\" (UniqueName: \"kubernetes.io/projected/f48c3035-122d-4c47-9310-fc188aab058b-kube-api-access-sknkf\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.762142 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.762124    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.762184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.762174    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.762832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.762811    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.764542 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.764519    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.818279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.818257    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sknkf\" (UniqueName: \"kubernetes.io/projected/f48c3035-122d-4c47-9310-fc188aab058b-kube-api-access-sknkf\") pod \"maas-api-revoke-sub-12dd1fb8-sjnqf\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:22.922919 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:22.922898    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:23.341136 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:23.338318    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"]
Nov 01 22:45:23.396066 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:23.384000    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48c3035_122d_4c47_9310_fc188aab058b.slice/crio-80637c60f301186305cf4dca99e4f1e6397f5da1e15436e4bd4d5667f162a840 WatchSource:0}: Error finding container 80637c60f301186305cf4dca99e4f1e6397f5da1e15436e4bd4d5667f162a840: Status 404 returned error can't find the container with id 80637c60f301186305cf4dca99e4f1e6397f5da1e15436e4bd4d5667f162a840
Nov 01 22:45:24.011154 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.011110    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"]
Nov 01 22:45:24.012440 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.012409    4837 cpu_manager.go:414] "containerMap: removing container" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:24.012639 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.012619    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:24.012818 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.012804    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:24.012943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.012928    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:24.013250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.013231    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4063873c-40dc-4bcb-9b5a-ef97d0728a53" containerName="cleanup"
Nov 01 22:45:24.013369 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.013357    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed8b004-81c4-40af-b8b7-8687b787c2de" containerName="cleanup"
Nov 01 22:45:24.013469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.013459    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="89281266-4a7a-4c49-90a6-6bcd3ea1677c" containerName="cleanup"
Nov 01 22:45:24.013547 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.013537    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" containerName="cleanup"
Nov 01 22:45:24.037024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.036994    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.069067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.058088    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"]
Nov 01 22:45:24.452599 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.452563    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.453185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.453160    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wvq\" (UniqueName: \"kubernetes.io/projected/a69654a5-8279-4298-b61d-8f56a47a4a79-kube-api-access-95wvq\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.453390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.453374    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.595066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.578907    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.595294 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.591603    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.681064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.637331    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.681365 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.681324    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wvq\" (UniqueName: \"kubernetes.io/projected/a69654a5-8279-4298-b61d-8f56a47a4a79-kube-api-access-95wvq\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.770068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.761956    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.814141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.812420    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b077949-67e7-4b28-89b5-9fa2dd34e0d7" path="/var/lib/kubelet/pods/2b077949-67e7-4b28-89b5-9fa2dd34e0d7/volumes"
Nov 01 22:45:24.828669 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.828548    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wvq\" (UniqueName: \"kubernetes.io/projected/a69654a5-8279-4298-b61d-8f56a47a4a79-kube-api-access-95wvq\") pod \"maas-api-revoke-sub-24113d11-4b4xl\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:24.837126 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.830324    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf" event={"ID":"f48c3035-122d-4c47-9310-fc188aab058b","Type":"ContainerStarted","Data":"80637c60f301186305cf4dca99e4f1e6397f5da1e15436e4bd4d5667f162a840"}
Nov 01 22:45:24.881064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.867997    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:24.891250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:24.891200    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.045062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.042127    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.072375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.072345    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:25.097414 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.085441    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.227088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.216475    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.239066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.235918    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 01 22:45:25.289026 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.288977    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.430196 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.430147    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.431644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.430464    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.832131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.817908    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:45:25.902237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:25.902189    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:45:26.130546 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.061253    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.137220 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.137192    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:45:26.145257 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.145234    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:45:26.289938 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.087326    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:45:26.467781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.205494    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.467781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.466894    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:45:26.467781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.276193    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.467781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.301132    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.467781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.302227    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.467781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.467237    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:45:26.779662 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.779565    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 22:45:26.830206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.830156    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.903110 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:26.851607    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565" podUID="87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.029075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.028294    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.156127 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.130222    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.236207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.157782    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.320527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.285442    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.553091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.526603    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.553091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.548980    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.553091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.549441    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.553091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.549759    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.553091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.550073    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.553091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.550387    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.667067 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:27.663509    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48c3035_122d_4c47_9310_fc188aab058b.slice/crio-c7847ebc08a8c8b14f26a268c8efe486cd5bbe910ffc12fb6fcae7b767fffa67.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:45:27.737551 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.729159    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.760596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.760559    4837 generic.go:356] "Generic (PLEG): container finished" podID="f48c3035-122d-4c47-9310-fc188aab058b" containerID="c7847ebc08a8c8b14f26a268c8efe486cd5bbe910ffc12fb6fcae7b767fffa67" exitCode=0
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.769086    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf" event={"ID":"f48c3035-122d-4c47-9310-fc188aab058b","Type":"ContainerDied","Data":"c7847ebc08a8c8b14f26a268c8efe486cd5bbe910ffc12fb6fcae7b767fffa67"}
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.769196    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.769240    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.769338    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.769366    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"]
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.769422    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.769452    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.769483    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:45:27.777073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.776179    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy" containerID="cri-o://86969515dc73b4da914bbe1108764e9072e421cfea4f3de89873dbc7055ca8e3" gracePeriod=30
Nov 01 22:45:27.782273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.782243    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.859129 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.821111    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"]
Nov 01 22:45:27.920081 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.915167    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"]
Nov 01 22:45:27.920081 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.915391    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" containerID="cri-o://054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0" gracePeriod=30
Nov 01 22:45:27.920081 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:27.915675    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy" containerID="cri-o://d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f" gracePeriod=30
Nov 01 22:45:28.184675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.184636    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Nov 01 22:45:28.248197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.248136    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.297078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.293767    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.352488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.352440    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.358507 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.353278    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.358951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.353999    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.359406 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.359367    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:45:28.563275 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.552580    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.584292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.584235    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.684963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.683534    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.734294 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.683681    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.734294 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.683712    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.793209 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:28.793164    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.038162 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:28.987666    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69654a5_8279_4298_b61d_8f56a47a4a79.slice/crio-conmon-0bec5378d807383805db7d087406fd8fd21dc151164d953dc335da7c3a07b258.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:45:29.193070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.144149    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.269266 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.254537    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.300483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.293964    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.374068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.368828    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"]
Nov 01 22:45:29.374068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.369090    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy" containerID="cri-o://00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4" gracePeriod=30
Nov 01 22:45:29.374068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.370357    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.374068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.371210    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.374068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.372383    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.486160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.477762    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl" event={"ID":"a69654a5-8279-4298-b61d-8f56a47a4a79","Type":"ContainerStarted","Data":"c61260cf87488c9581e67656f1c3bb78120b9abcba211245600b0f43ef16e69f"}
Nov 01 22:45:29.553312 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.553197    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.811732 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.811621    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:30.072541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.903789    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.072541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.903923    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.072541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.904022    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.072541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.958810    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.072541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:29.988684    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"]
Nov 01 22:45:30.428757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:30.395438    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:30.487457 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:30.487408    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.662453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:30.662230    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:30.722905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:30.722804    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.796032 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:30.794498    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 01 22:45:30.802265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:30.802218    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.971685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:30.963204    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:45:31.110119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.102449    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 22:45:31.204426 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.113299    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:31.208589 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.113353    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:45:31.359634 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.113397    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:45:31.360567 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.277292    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565" podUID="87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:31.424203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424168    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl" event={"ID":"a69654a5-8279-4298-b61d-8f56a47a4a79","Type":"ContainerStarted","Data":"0bec5378d807383805db7d087406fd8fd21dc151164d953dc335da7c3a07b258"}
Nov 01 22:45:31.424383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424240    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:45:31.424383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424264    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:45:31.424383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424292    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:45:31.424383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424311    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:45:31.424383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424332    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:45:31.424383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424354    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:45:31.424383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424375    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:45:31.424749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424394    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:45:31.424749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424412    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:45:31.424749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424433    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:45:31.424749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.424454    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:45:31.427836 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.427811    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:31.522779 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.522750    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-xks-ca\") pod \"f48c3035-122d-4c47-9310-fc188aab058b\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") "
Nov 01 22:45:31.522940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.522806    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48c3035-122d-4c47-9310-fc188aab058b-kube-api-access-sknkf\" (UniqueName: \"kubernetes.io/projected/f48c3035-122d-4c47-9310-fc188aab058b-kube-api-access-sknkf\") pod \"f48c3035-122d-4c47-9310-fc188aab058b\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") "
Nov 01 22:45:31.522940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.522916    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-openshift-ca\") pod \"f48c3035-122d-4c47-9310-fc188aab058b\" (UID: \"f48c3035-122d-4c47-9310-fc188aab058b\") "
Nov 01 22:45:31.523320 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.523290    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-openshift-ca" pod "f48c3035-122d-4c47-9310-fc188aab058b" (UID: "f48c3035-122d-4c47-9310-fc188aab058b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:31.524233 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.524207    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:31.525232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.525208    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-xks-ca" pod "f48c3035-122d-4c47-9310-fc188aab058b" (UID: "f48c3035-122d-4c47-9310-fc188aab058b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:31.525965 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.525949    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48c3035-122d-4c47-9310-fc188aab058b-kube-api-access-sknkf" pod "f48c3035-122d-4c47-9310-fc188aab058b" (UID: "f48c3035-122d-4c47-9310-fc188aab058b"). InnerVolumeSpecName "kube-api-access-sknkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:31.625898 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.625822    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f48c3035-122d-4c47-9310-fc188aab058b-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:31.626097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:31.626077    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sknkf\" (UniqueName: \"kubernetes.io/projected/f48c3035-122d-4c47-9310-fc188aab058b-kube-api-access-sknkf\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:32.435959 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.435926    4837 generic.go:356] "Generic (PLEG): container finished" podID="a69654a5-8279-4298-b61d-8f56a47a4a79" containerID="0bec5378d807383805db7d087406fd8fd21dc151164d953dc335da7c3a07b258" exitCode=0
Nov 01 22:45:32.436434 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.436006    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl" event={"ID":"a69654a5-8279-4298-b61d-8f56a47a4a79","Type":"ContainerDied","Data":"0bec5378d807383805db7d087406fd8fd21dc151164d953dc335da7c3a07b258"}
Nov 01 22:45:32.437638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.437609    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"
Nov 01 22:45:32.437729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.437650    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf" event={"ID":"f48c3035-122d-4c47-9310-fc188aab058b","Type":"ContainerDied","Data":"80637c60f301186305cf4dca99e4f1e6397f5da1e15436e4bd4d5667f162a840"}
Nov 01 22:45:32.437729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.437709    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80637c60f301186305cf4dca99e4f1e6397f5da1e15436e4bd4d5667f162a840"
Nov 01 22:45:32.609029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.609000    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:32.659853 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.659830    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-xks-ca\") pod \"a69654a5-8279-4298-b61d-8f56a47a4a79\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") "
Nov 01 22:45:32.659987 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.659912    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69654a5-8279-4298-b61d-8f56a47a4a79-kube-api-access-95wvq\" (UniqueName: \"kubernetes.io/projected/a69654a5-8279-4298-b61d-8f56a47a4a79-kube-api-access-95wvq\") pod \"a69654a5-8279-4298-b61d-8f56a47a4a79\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") "
Nov 01 22:45:32.660060 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.660001    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-openshift-ca\") pod \"a69654a5-8279-4298-b61d-8f56a47a4a79\" (UID: \"a69654a5-8279-4298-b61d-8f56a47a4a79\") "
Nov 01 22:45:32.660327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.660303    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-openshift-ca" pod "a69654a5-8279-4298-b61d-8f56a47a4a79" (UID: "a69654a5-8279-4298-b61d-8f56a47a4a79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:32.660586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.660569    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:32.661930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.661910    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69654a5-8279-4298-b61d-8f56a47a4a79-kube-api-access-95wvq" pod "a69654a5-8279-4298-b61d-8f56a47a4a79" (UID: "a69654a5-8279-4298-b61d-8f56a47a4a79"). InnerVolumeSpecName "kube-api-access-95wvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:32.662003 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.661957    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-xks-ca" pod "a69654a5-8279-4298-b61d-8f56a47a4a79" (UID: "a69654a5-8279-4298-b61d-8f56a47a4a79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:32.762729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.761819    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69654a5-8279-4298-b61d-8f56a47a4a79-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:32.762729 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:32.761852    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95wvq\" (UniqueName: \"kubernetes.io/projected/a69654a5-8279-4298-b61d-8f56a47a4a79-kube-api-access-95wvq\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.451161 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.451088    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rsr62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:45:33.451708 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.451661    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:45:33.457752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.457723    4837 generic.go:356] "Generic (PLEG): container finished" podID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerID="86969515dc73b4da914bbe1108764e9072e421cfea4f3de89873dbc7055ca8e3" exitCode=0
Nov 01 22:45:33.457916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.457897    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" event={"ID":"f07815f2-3a45-405c-a1cc-1a1f703c0e5c","Type":"ContainerDied","Data":"86969515dc73b4da914bbe1108764e9072e421cfea4f3de89873dbc7055ca8e3"}
Nov 01 22:45:33.459847 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.459825    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"
Nov 01 22:45:33.459969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.459830    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl" event={"ID":"a69654a5-8279-4298-b61d-8f56a47a4a79","Type":"ContainerDied","Data":"c61260cf87488c9581e67656f1c3bb78120b9abcba211245600b0f43ef16e69f"}
Nov 01 22:45:33.459969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.459943    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c61260cf87488c9581e67656f1c3bb78120b9abcba211245600b0f43ef16e69f"
Nov 01 22:45:33.576797 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.575868    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:45:33.590893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.590864    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-kube-api-access-bndhn\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-kube-api-access-bndhn\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591036 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.590926    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istiod-ca-cert\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591036 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.590991    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-certs\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591189 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.591083    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-credential-socket\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591189 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.591113    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-data\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591189 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.591156    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-podinfo\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.591228    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-token\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-token\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.591250    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-envoy\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.591313    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-socket\") pod \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\" (UID: \"f07815f2-3a45-405c-a1cc-1a1f703c0e5c\") "
Nov 01 22:45:33.591922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.591894    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-socket" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.592133 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.592090    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-credential-socket" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.592224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.592167    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-data" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.594962 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.594931    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-podinfo" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:45:33.595314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.595285    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-certs" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.595397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.595351    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istiod-ca-cert" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:33.597131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.597055    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-kube-api-access-bndhn" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "kube-api-access-bndhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.597803 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.597777    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-token" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.600206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.600180    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-envoy" pod "f07815f2-3a45-405c-a1cc-1a1f703c0e5c" (UID: "f07815f2-3a45-405c-a1cc-1a1f703c0e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.693193 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693158    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.693193 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693188    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.693193 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693197    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.693469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693205    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.693469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693214    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.693469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693222    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.693469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693231    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bndhn\" (UniqueName: \"kubernetes.io/projected/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-kube-api-access-bndhn\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.693469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693244    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.693469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.693258    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07815f2-3a45-405c-a1cc-1a1f703c0e5c-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.775972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.775945    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:45:33.794431 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794405    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-kube-api-access-p84xg\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-kube-api-access-p84xg\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.794556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794474    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabfcadb-8c03-4339-8129-7814d204dfe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfcadb-8c03-4339-8129-7814d204dfe2-istiod-ca-cert\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.794685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794567    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-socket\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.794747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794729    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-credential-socket\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.794845 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794813    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-envoy\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.794904 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794879    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-socket" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.794980 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794915    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-certs\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.794980 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794970    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-data\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.795201 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.794980    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabfcadb-8c03-4339-8129-7814d204dfe2-istiod-ca-cert" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:33.795201 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795011    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-podinfo\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.795201 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795097    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-token\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-token\") pod \"cabfcadb-8c03-4339-8129-7814d204dfe2\" (UID: \"cabfcadb-8c03-4339-8129-7814d204dfe2\") "
Nov 01 22:45:33.795358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795214    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-certs" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.795358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795278    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-credential-socket" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.795358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795294    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-data" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.795719 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795698    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.795800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795725    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.795800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795740    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfcadb-8c03-4339-8129-7814d204dfe2-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.795800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795755    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.795800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.795770    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.797807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.797782    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-token" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.797807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.797782    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-podinfo" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:45:33.797982 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.797962    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-envoy" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.810736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.802370    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-kube-api-access-p84xg" pod "cabfcadb-8c03-4339-8129-7814d204dfe2" (UID: "cabfcadb-8c03-4339-8129-7814d204dfe2"). InnerVolumeSpecName "kube-api-access-p84xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.833038 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.833018    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:45:33.896906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.896872    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-token\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-token\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.896928    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-credential-socket\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897027    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-socket\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897105    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-envoy\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897151    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-podinfo\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897333 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897237    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-credential-socket" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.897333 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897270    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-kube-api-access-nzk87\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-kube-api-access-nzk87\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897333 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897304    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-certs\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897335    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-data\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897415    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955785c3-9b45-4da1-ae35-63861a17b4a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955785c3-9b45-4da1-ae35-63861a17b4a3-istiod-ca-cert\") pod \"955785c3-9b45-4da1-ae35-63861a17b4a3\" (UID: \"955785c3-9b45-4da1-ae35-63861a17b4a3\") "
Nov 01 22:45:33.897621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897581    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-data" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.897849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897823    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-certs" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.898272 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.898252    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.898372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.898277    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p84xg\" (UniqueName: \"kubernetes.io/projected/cabfcadb-8c03-4339-8129-7814d204dfe2-kube-api-access-p84xg\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.898372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.898296    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.898372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.898308    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.898372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.898321    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.898372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.898334    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.898372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.898348    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfcadb-8c03-4339-8129-7814d204dfe2-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.898731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.897912    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955785c3-9b45-4da1-ae35-63861a17b4a3-istiod-ca-cert" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:33.899206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.899168    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-socket" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.900640 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.900611    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-token" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.900746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.900628    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-podinfo" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:45:33.900936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.900909    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-envoy" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.902109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:33.902027    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-kube-api-access-nzk87" pod "955785c3-9b45-4da1-ae35-63861a17b4a3" (UID: "955785c3-9b45-4da1-ae35-63861a17b4a3"). InnerVolumeSpecName "kube-api-access-nzk87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:34.002699 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.002605    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzk87\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-kube-api-access-nzk87\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.002699 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.002635    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955785c3-9b45-4da1-ae35-63861a17b4a3-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.002699 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.002649    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.002699 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.002663    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.002699 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.002676    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.002699 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.002687    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955785c3-9b45-4da1-ae35-63861a17b4a3-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.470973 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.470933    4837 generic.go:356] "Generic (PLEG): container finished" podID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerID="054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0" exitCode=0
Nov 01 22:45:34.471568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.470982    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"
Nov 01 22:45:34.471568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.471007    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" event={"ID":"955785c3-9b45-4da1-ae35-63861a17b4a3","Type":"ContainerDied","Data":"054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0"}
Nov 01 22:45:34.471568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.471081    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62" event={"ID":"955785c3-9b45-4da1-ae35-63861a17b4a3","Type":"ContainerDied","Data":"63d6c01ad72fd0ca71e02bf87957edff7b1150387bc0ad8d8ecdae9614c6fba3"}
Nov 01 22:45:34.471568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.471114    4837 scope.go:122] "RemoveContainer" containerID="054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0"
Nov 01 22:45:34.473100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.473075    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j" event={"ID":"f07815f2-3a45-405c-a1cc-1a1f703c0e5c","Type":"ContainerDied","Data":"01f9b59edb034afbea99420845d000d2f114d15cc4e3ebc64991e42038dce798"}
Nov 01 22:45:34.473492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.473467    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"
Nov 01 22:45:34.479683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.479661    4837 generic.go:356] "Generic (PLEG): container finished" podID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerID="d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f" exitCode=0
Nov 01 22:45:34.479761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.479706    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"
Nov 01 22:45:34.479813 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.479792    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" event={"ID":"cabfcadb-8c03-4339-8129-7814d204dfe2","Type":"ContainerDied","Data":"d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f"}
Nov 01 22:45:34.479866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.479844    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js" event={"ID":"cabfcadb-8c03-4339-8129-7814d204dfe2","Type":"ContainerDied","Data":"a300cd0f8c2fc00f45735cf2e176d194adc292fffa58c6228f10112b50a58f45"}
Nov 01 22:45:34.485413 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.485394    4837 scope.go:122] "RemoveContainer" containerID="054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0"
Nov 01 22:45:34.490329 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:34.486130    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0\": container with ID starting with 054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0 not found: ID does not exist" containerID="054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0"
Nov 01 22:45:34.490329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.486180    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0"} err="failed to get container status \"054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0\": rpc error: code = NotFound desc = could not find container \"054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0\": container with ID starting with 054777001cc6f37a25c27484878379a04155b047a5d580c61a5f8ae75cefe5a0 not found: ID does not exist"
Nov 01 22:45:34.490329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.486204    4837 scope.go:122] "RemoveContainer" containerID="86969515dc73b4da914bbe1108764e9072e421cfea4f3de89873dbc7055ca8e3"
Nov 01 22:45:34.498579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.498563    4837 scope.go:122] "RemoveContainer" containerID="d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f"
Nov 01 22:45:34.507746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.507727    4837 scope.go:122] "RemoveContainer" containerID="d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f"
Nov 01 22:45:34.508000 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:34.507984    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f\": container with ID starting with d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f not found: ID does not exist" containerID="d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f"
Nov 01 22:45:34.508082 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.508005    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f"} err="failed to get container status \"d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f\": rpc error: code = NotFound desc = could not find container \"d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f\": container with ID starting with d35307bb3950fd7ddc3e9de22f1cb91ea6e0c83d4998ee16a71233e01418e03f not found: ID does not exist"
Nov 01 22:45:34.630279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.630254    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"]
Nov 01 22:45:34.636675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.636649    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rsr62"]
Nov 01 22:45:34.742511 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.742483    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"]
Nov 01 22:45:34.748468 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.748440    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ggm2j"]
Nov 01 22:45:34.848823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.848795    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"]
Nov 01 22:45:34.854078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.854023    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tm4js"]
Nov 01 22:45:34.886868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.886846    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:45:34.932529 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932501    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-credential-socket\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.932657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932558    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-data\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.932657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932620    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-socket\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.932781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932668    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-istio-token\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-istio-token\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.932781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932707    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-certs\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.932781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932746    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-credential-socket" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:34.932951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932800    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1a58aa0-2864-43d4-b7be-60228b973837-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1a58aa0-2864-43d4-b7be-60228b973837-istio-podinfo\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.932951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932835    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-data" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:34.932951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932857    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-kube-api-access-z9xr5\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-kube-api-access-z9xr5\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.932951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932923    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-socket" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:34.933151 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932948    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a58aa0-2864-43d4-b7be-60228b973837-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1a58aa0-2864-43d4-b7be-60228b973837-istiod-ca-cert\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.933151 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.932996    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-envoy\") pod \"c1a58aa0-2864-43d4-b7be-60228b973837\" (UID: \"c1a58aa0-2864-43d4-b7be-60228b973837\") "
Nov 01 22:45:34.933151 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.933034    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-certs" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:34.933342 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.933317    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a58aa0-2864-43d4-b7be-60228b973837-istiod-ca-cert" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:34.933625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.933605    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.933625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.933631    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.933752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.933646    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1a58aa0-2864-43d4-b7be-60228b973837-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.933752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.933660    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.933752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.933675    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.934967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.934946    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1a58aa0-2864-43d4-b7be-60228b973837-istio-podinfo" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:45:34.935476 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.935449    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-envoy" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:34.935558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.935516    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-kube-api-access-z9xr5" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "kube-api-access-z9xr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:34.935558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:34.935536    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-istio-token" pod "c1a58aa0-2864-43d4-b7be-60228b973837" (UID: "c1a58aa0-2864-43d4-b7be-60228b973837"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:35.034579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.034498    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9xr5\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-kube-api-access-z9xr5\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:35.034579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.034536    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1a58aa0-2864-43d4-b7be-60228b973837-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:35.034579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.034546    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1a58aa0-2864-43d4-b7be-60228b973837-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:35.034579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.034554    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1a58aa0-2864-43d4-b7be-60228b973837-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:35.491061 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.491016    4837 generic.go:356] "Generic (PLEG): container finished" podID="c1a58aa0-2864-43d4-b7be-60228b973837" containerID="00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4" exitCode=0
Nov 01 22:45:35.491500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.491099    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"
Nov 01 22:45:35.491500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.491095    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" event={"ID":"c1a58aa0-2864-43d4-b7be-60228b973837","Type":"ContainerDied","Data":"00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4"}
Nov 01 22:45:35.491500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.491212    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6" event={"ID":"c1a58aa0-2864-43d4-b7be-60228b973837","Type":"ContainerDied","Data":"f3afaa7b6c6f90d261f533d29c3ebcdadb3616cd6964786fbc01cdc0ad606805"}
Nov 01 22:45:35.491500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.491247    4837 scope.go:122] "RemoveContainer" containerID="00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4"
Nov 01 22:45:35.501906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.501890    4837 scope.go:122] "RemoveContainer" containerID="00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4"
Nov 01 22:45:35.502201 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:35.502183    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4\": container with ID starting with 00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4 not found: ID does not exist" containerID="00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4"
Nov 01 22:45:35.502246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.502211    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4"} err="failed to get container status \"00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4\": rpc error: code = NotFound desc = could not find container \"00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4\": container with ID starting with 00b3ad92a0e977a257328c0686741cc8e818d7967a9e621731a06da9f30cb5e4 not found: ID does not exist"
Nov 01 22:45:35.665110 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.665074    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"]
Nov 01 22:45:35.681859 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:35.681836    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cp2r6"]
Nov 01 22:45:36.525208 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:36.525169    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" path="/var/lib/kubelet/pods/955785c3-9b45-4da1-ae35-63861a17b4a3/volumes"
Nov 01 22:45:36.525904 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:36.525885    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" path="/var/lib/kubelet/pods/c1a58aa0-2864-43d4-b7be-60228b973837/volumes"
Nov 01 22:45:36.526365 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:36.526350    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" path="/var/lib/kubelet/pods/cabfcadb-8c03-4339-8129-7814d204dfe2/volumes"
Nov 01 22:45:36.527133 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:36.527118    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" path="/var/lib/kubelet/pods/f07815f2-3a45-405c-a1cc-1a1f703c0e5c/volumes"
Nov 01 22:45:37.646693 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.646663    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"]
Nov 01 22:45:37.647279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647264    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f48c3035-122d-4c47-9310-fc188aab058b" containerName="revoke-keys"
Nov 01 22:45:37.647331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647281    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48c3035-122d-4c47-9310-fc188aab058b" containerName="revoke-keys"
Nov 01 22:45:37.647331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647302    4837 cpu_manager.go:414] "containerMap: removing container" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy"
Nov 01 22:45:37.647331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647307    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy"
Nov 01 22:45:37.647331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647318    4837 cpu_manager.go:414] "containerMap: removing container" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy"
Nov 01 22:45:37.647331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647323    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy"
Nov 01 22:45:37.647472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647341    4837 cpu_manager.go:414] "containerMap: removing container" podUID="a69654a5-8279-4298-b61d-8f56a47a4a79" containerName="revoke-keys"
Nov 01 22:45:37.647472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647346    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69654a5-8279-4298-b61d-8f56a47a4a79" containerName="revoke-keys"
Nov 01 22:45:37.647472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647357    4837 cpu_manager.go:414] "containerMap: removing container" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy"
Nov 01 22:45:37.647472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647361    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy"
Nov 01 22:45:37.647472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647372    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy"
Nov 01 22:45:37.647472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647376    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy"
Nov 01 22:45:37.647472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647468    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabfcadb-8c03-4339-8129-7814d204dfe2" containerName="istio-proxy"
Nov 01 22:45:37.647657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647476    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a58aa0-2864-43d4-b7be-60228b973837" containerName="istio-proxy"
Nov 01 22:45:37.647657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647484    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69654a5-8279-4298-b61d-8f56a47a4a79" containerName="revoke-keys"
Nov 01 22:45:37.647657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647491    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="955785c3-9b45-4da1-ae35-63861a17b4a3" containerName="istio-proxy"
Nov 01 22:45:37.647657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647500    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07815f2-3a45-405c-a1cc-1a1f703c0e5c" containerName="istio-proxy"
Nov 01 22:45:37.647657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.647507    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48c3035-122d-4c47-9310-fc188aab058b" containerName="revoke-keys"
Nov 01 22:45:37.652384 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.652369    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.673332 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.673313    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.673537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.673515    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.673651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.673637    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2xn\" (UniqueName: \"kubernetes.io/projected/35e7028a-755b-435b-a3a7-338b93cd6132-kube-api-access-bq2xn\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.702254 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.702233    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:37.748488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.748462    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:45:37.752565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.752541    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"]
Nov 01 22:45:37.774918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.774894    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.775016 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.774956    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2xn\" (UniqueName: \"kubernetes.io/projected/35e7028a-755b-435b-a3a7-338b93cd6132-kube-api-access-bq2xn\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.775112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.775026    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.775610 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.775592    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.777566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.777539    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.830280 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.830260    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2xn\" (UniqueName: \"kubernetes.io/projected/35e7028a-755b-435b-a3a7-338b93cd6132-kube-api-access-bq2xn\") pod \"maas-api-revoke-sub-1152c744-rsnrh\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:37.966619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:37.966594    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:38.342884 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:38.342856    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e7028a_755b_435b_a3a7_338b93cd6132.slice/crio-a095540d9adcc883f70cef3e633df3b421d3b08a0ba81741ec6e98b74af9f3a3 WatchSource:0}: Error finding container a095540d9adcc883f70cef3e633df3b421d3b08a0ba81741ec6e98b74af9f3a3: Status 404 returned error can't find the container with id a095540d9adcc883f70cef3e633df3b421d3b08a0ba81741ec6e98b74af9f3a3
Nov 01 22:45:38.343324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:38.343295    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"]
Nov 01 22:45:38.515184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:38.515147    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh" event={"ID":"35e7028a-755b-435b-a3a7-338b93cd6132","Type":"ContainerStarted","Data":"a095540d9adcc883f70cef3e633df3b421d3b08a0ba81741ec6e98b74af9f3a3"}
Nov 01 22:45:38.677254 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:38.677228    4837 scope.go:122] "RemoveContainer" containerID="8970f2e226792dff33cba56f5f44577211b09343f8da6af053d3cb07d879fe02"
Nov 01 22:45:39.523129 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:39.523021    4837 generic.go:356] "Generic (PLEG): container finished" podID="35e7028a-755b-435b-a3a7-338b93cd6132" containerID="f456f6dc6d3718f7845e90fd386a30f203824befd91b0eea26356581e481839d" exitCode=0
Nov 01 22:45:39.523129 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:39.523109    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh" event={"ID":"35e7028a-755b-435b-a3a7-338b93cd6132","Type":"ContainerDied","Data":"f456f6dc6d3718f7845e90fd386a30f203824befd91b0eea26356581e481839d"}
Nov 01 22:45:40.717037 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.717014    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:40.813954 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.813898    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-openshift-ca\") pod \"35e7028a-755b-435b-a3a7-338b93cd6132\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") "
Nov 01 22:45:40.814087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.813986    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-xks-ca\") pod \"35e7028a-755b-435b-a3a7-338b93cd6132\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") "
Nov 01 22:45:40.814087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.814070    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e7028a-755b-435b-a3a7-338b93cd6132-kube-api-access-bq2xn\" (UniqueName: \"kubernetes.io/projected/35e7028a-755b-435b-a3a7-338b93cd6132-kube-api-access-bq2xn\") pod \"35e7028a-755b-435b-a3a7-338b93cd6132\" (UID: \"35e7028a-755b-435b-a3a7-338b93cd6132\") "
Nov 01 22:45:40.814322 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.814296    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-openshift-ca" pod "35e7028a-755b-435b-a3a7-338b93cd6132" (UID: "35e7028a-755b-435b-a3a7-338b93cd6132"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:40.814498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.814483    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:40.816091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.816038    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e7028a-755b-435b-a3a7-338b93cd6132-kube-api-access-bq2xn" pod "35e7028a-755b-435b-a3a7-338b93cd6132" (UID: "35e7028a-755b-435b-a3a7-338b93cd6132"). InnerVolumeSpecName "kube-api-access-bq2xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:40.816217 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.816155    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-xks-ca" pod "35e7028a-755b-435b-a3a7-338b93cd6132" (UID: "35e7028a-755b-435b-a3a7-338b93cd6132"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:40.915519 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.915490    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e7028a-755b-435b-a3a7-338b93cd6132-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:40.915519 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:40.915517    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq2xn\" (UniqueName: \"kubernetes.io/projected/35e7028a-755b-435b-a3a7-338b93cd6132-kube-api-access-bq2xn\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:41.565261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:41.565228    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"
Nov 01 22:45:41.565447 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:41.565265    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh" event={"ID":"35e7028a-755b-435b-a3a7-338b93cd6132","Type":"ContainerDied","Data":"a095540d9adcc883f70cef3e633df3b421d3b08a0ba81741ec6e98b74af9f3a3"}
Nov 01 22:45:41.565447 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:41.565320    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a095540d9adcc883f70cef3e633df3b421d3b08a0ba81741ec6e98b74af9f3a3"
Nov 01 22:45:41.924075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:41.911427    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"]
Nov 01 22:45:41.924626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:41.924352    4837 cpu_manager.go:414] "containerMap: removing container" podUID="35e7028a-755b-435b-a3a7-338b93cd6132" containerName="revoke-keys"
Nov 01 22:45:41.924626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:41.924376    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e7028a-755b-435b-a3a7-338b93cd6132" containerName="revoke-keys"
Nov 01 22:45:41.924626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:41.924548    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e7028a-755b-435b-a3a7-338b93cd6132" containerName="revoke-keys"
Nov 01 22:45:42.225500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.225417    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.286004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.285974    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d5014777-openshift-default-dockercfg-tcmp5\""
Nov 01 22:45:42.291955 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.287280    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"]
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.330794    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-credential-socket\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.330840    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-certs\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.330865    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-envoy\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.330980    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblgz\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-kube-api-access-cblgz\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.331109    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c518274-3546-42b4-8da0-04fcb26802bf-istiod-ca-cert\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.331200    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-istio-token\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.331263    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-socket\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.331287    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c518274-3546-42b4-8da0-04fcb26802bf-istio-podinfo\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.332486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.331328    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-data\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.432516 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432490    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-credential-socket\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.432683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432526    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-certs\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.432683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432557    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-envoy\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.432683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432594    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblgz\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-kube-api-access-cblgz\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.432683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432676    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c518274-3546-42b4-8da0-04fcb26802bf-istiod-ca-cert\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432762    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-istio-token\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432813    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-socket\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432836    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c518274-3546-42b4-8da0-04fcb26802bf-istio-podinfo\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.432883    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-data\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.433318    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-socket\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.433817    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-credential-socket\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.434491    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-data\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.436147    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c518274-3546-42b4-8da0-04fcb26802bf-istio-podinfo\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.436306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.436225    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-envoy\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.461282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.446337    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-certs\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.461282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.450300    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c518274-3546-42b4-8da0-04fcb26802bf-istiod-ca-cert\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.498866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.495416    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblgz\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-kube-api-access-cblgz\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.503872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.499610    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-istio-token\") pod \"e2e-ait-d5014777-openshift-default-644c7559cf-9grq7\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") " pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:42.604582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:42.604544    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:43.039966 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.039938    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"]
Nov 01 22:45:43.061552 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:43.061524    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c518274_3546_42b4_8da0_04fcb26802bf.slice/crio-8b36a2b0ffb76f5963bdea43830313cc747542600787dac9395fb588ec32ae09 WatchSource:0}: Error finding container 8b36a2b0ffb76f5963bdea43830313cc747542600787dac9395fb588ec32ae09: Status 404 returned error can't find the container with id 8b36a2b0ffb76f5963bdea43830313cc747542600787dac9395fb588ec32ae09
Nov 01 22:45:43.063942 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.063908    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:43.064024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.063980    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:43.064024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.064013    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:43.534496 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.534470    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"]
Nov 01 22:45:43.538869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.538850    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.581138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.581106    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" event={"ID":"0c518274-3546-42b4-8da0-04fcb26802bf","Type":"ContainerStarted","Data":"fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751"}
Nov 01 22:45:43.581265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.581175    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" event={"ID":"0c518274-3546-42b4-8da0-04fcb26802bf","Type":"ContainerStarted","Data":"8b36a2b0ffb76f5963bdea43830313cc747542600787dac9395fb588ec32ae09"}
Nov 01 22:45:43.589187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.589168    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:43.589282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.589245    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:45:43.604735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.604713    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:43.640803 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.640773    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"]
Nov 01 22:45:43.654003 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.653974    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9srb\" (UniqueName: \"kubernetes.io/projected/97581061-e864-4439-95d2-458e444b18cc-kube-api-access-p9srb\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.654146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.654061    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97581061-e864-4439-95d2-458e444b18cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.654283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.654267    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97581061-e864-4439-95d2-458e444b18cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.756234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.756206    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9srb\" (UniqueName: \"kubernetes.io/projected/97581061-e864-4439-95d2-458e444b18cc-kube-api-access-p9srb\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.756377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.756258    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97581061-e864-4439-95d2-458e444b18cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.756377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.756326    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97581061-e864-4439-95d2-458e444b18cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.756937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.756915    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97581061-e864-4439-95d2-458e444b18cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.758661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.758638    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97581061-e864-4439-95d2-458e444b18cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.828391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.828299    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podStartSLOduration=2.828285069 podStartE2EDuration="2.828285069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:43.823948135 +0000 UTC m=+1325.413147199" watchObservedRunningTime="2026-11-01 22:45:43.828285069 +0000 UTC m=+1325.417484130"
Nov 01 22:45:43.829370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.829347    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9srb\" (UniqueName: \"kubernetes.io/projected/97581061-e864-4439-95d2-458e444b18cc-kube-api-access-p9srb\") pod \"maas-api-revoke-sub-997e8a96-8pcht\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:43.855724 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:43.855704    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:44.303989 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:44.291102    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"]
Nov 01 22:45:44.678063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:44.675097    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:44.684371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:44.682519    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht" event={"ID":"97581061-e864-4439-95d2-458e444b18cc","Type":"ContainerStarted","Data":"1c899cb8ebc46da6fb06742a1cfd6eec54ea635842aa82c3c34e1ec2c6347652"}
Nov 01 22:45:44.684371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:44.682578    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:44.684371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:44.684283    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:45:45.193178 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.193146    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"]
Nov 01 22:45:45.197533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.197514    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.212580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.212556    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.212686 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.212666    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbtd\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-kube-api-access-7lbtd\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.212728 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.212702    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.212767 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.212742    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468199b3-71a0-4e45-927f-3dbe77075f56-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.212895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.212876    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468199b3-71a0-4e45-927f-3dbe77075f56-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.212932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.212911    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.212985 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.212937    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.213026 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.212981    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.213071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.213030    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.284290 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.284261    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"]
Nov 01 22:45:45.314156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.314117    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315101 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315035    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.314951    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315200    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbtd\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-kube-api-access-7lbtd\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315341 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315234    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315341 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315288    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468199b3-71a0-4e45-927f-3dbe77075f56-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315390    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468199b3-71a0-4e45-927f-3dbe77075f56-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315427    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315459    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315558 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315482    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315608    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315790 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315718    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.315862 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.315841    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.316355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.316333    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468199b3-71a0-4e45-927f-3dbe77075f56-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.317799 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.317776    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.318111 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.318091    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468199b3-71a0-4e45-927f-3dbe77075f56-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.373951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.373925    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.374835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.374816    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbtd\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-kube-api-access-7lbtd\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.511290 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.511223    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:45.527571 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.527545    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"]
Nov 01 22:45:45.533007 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.532988    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.584923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.584884    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-798f4550-openshift-default-dockercfg-ltxzz\""
Nov 01 22:45:45.602648 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.602544    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"]
Nov 01 22:45:45.619578 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.619544    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-data\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.619712 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.619594    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-socket\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.619766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.619744    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2rh\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-kube-api-access-jd2rh\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.619808 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.619790    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d0c78-8fc8-4f44-9b93-96951093c210-istiod-ca-cert\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.619883 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.619863    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-envoy\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.619936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.619924    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-credential-socket\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.620008 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.619996    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-istio-token\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.620160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.620141    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-certs\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.620221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.620187    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d0c78-8fc8-4f44-9b93-96951093c210-istio-podinfo\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.691689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.691652    4837 generic.go:356] "Generic (PLEG): container finished" podID="97581061-e864-4439-95d2-458e444b18cc" containerID="4d94fed2561ef71d7e17345727083e45733d9d6d176ac59db7278461061118da" exitCode=0
Nov 01 22:45:45.691856 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.691732    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht" event={"ID":"97581061-e864-4439-95d2-458e444b18cc","Type":"ContainerDied","Data":"4d94fed2561ef71d7e17345727083e45733d9d6d176ac59db7278461061118da"}
Nov 01 22:45:45.722069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722009    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-certs\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722094    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d0c78-8fc8-4f44-9b93-96951093c210-istio-podinfo\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722165    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-data\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722186    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-socket\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722409 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722250    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2rh\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-kube-api-access-jd2rh\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722409 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722285    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d0c78-8fc8-4f44-9b93-96951093c210-istiod-ca-cert\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722511    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-certs\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722597 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722552    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-data\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722610    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-socket\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722709 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722695    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-envoy\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722737    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-credential-socket\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.722870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722854    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-istio-token\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.723007 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.722981    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d0c78-8fc8-4f44-9b93-96951093c210-istiod-ca-cert\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.723297 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.723279    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-credential-socket\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.725084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.725038    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-envoy\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.725830 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.725808    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d0c78-8fc8-4f44-9b93-96951093c210-istio-podinfo\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.829162 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.816662    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2rh\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-kube-api-access-jd2rh\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.856744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.856669    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-istio-token\") pod \"e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") " pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:45.868587 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:45.868312    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:46.069616 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:46.068777    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12 WatchSource:0}: Error finding container 76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12: Status 404 returned error can't find the container with id 76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12
Nov 01 22:45:46.069616 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.069385    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"]
Nov 01 22:45:46.072200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.072167    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:46.072422 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.072395    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:46.072591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.072571    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:46.358638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.358613    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"]
Nov 01 22:45:46.359701 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:46.359676    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600d0c78_8fc8_4f44_9b93_96951093c210.slice/crio-5bd1d6f7266fc21aa338b15bb943b3a4323d8b06614704edf878dc1a0951735d WatchSource:0}: Error finding container 5bd1d6f7266fc21aa338b15bb943b3a4323d8b06614704edf878dc1a0951735d: Status 404 returned error can't find the container with id 5bd1d6f7266fc21aa338b15bb943b3a4323d8b06614704edf878dc1a0951735d
Nov 01 22:45:46.362542 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.362506    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:46.362661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.362584    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:46.362661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.362627    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:46.702139 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.702100    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" event={"ID":"468199b3-71a0-4e45-927f-3dbe77075f56","Type":"ContainerStarted","Data":"c385554dff7fdcb7278617177272e1ca4da51ce80898e3e371d340fadd4f4ec3"}
Nov 01 22:45:46.702351 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.702188    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" event={"ID":"468199b3-71a0-4e45-927f-3dbe77075f56","Type":"ContainerStarted","Data":"76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12"}
Nov 01 22:45:46.703855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.703826    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" event={"ID":"600d0c78-8fc8-4f44-9b93-96951093c210","Type":"ContainerStarted","Data":"a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab"}
Nov 01 22:45:46.703993 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.703875    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" event={"ID":"600d0c78-8fc8-4f44-9b93-96951093c210","Type":"ContainerStarted","Data":"5bd1d6f7266fc21aa338b15bb943b3a4323d8b06614704edf878dc1a0951735d"}
Nov 01 22:45:46.889344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.871269    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podStartSLOduration=1.871247646 podStartE2EDuration="1.871247646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:46.829351152 +0000 UTC m=+1328.418550240" watchObservedRunningTime="2026-11-01 22:45:46.871247646 +0000 UTC m=+1328.460446708"
Nov 01 22:45:46.889344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.874327    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:46.963205 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.958376    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:46.963205 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.961134    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" podStartSLOduration=1.96111559 podStartE2EDuration="1.96111559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:46.954943644 +0000 UTC m=+1328.544142709" watchObservedRunningTime="2026-11-01 22:45:46.96111559 +0000 UTC m=+1328.550314660"
Nov 01 22:45:46.974897 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.974868    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97581061-e864-4439-95d2-458e444b18cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97581061-e864-4439-95d2-458e444b18cc-maas-api-xks-ca\") pod \"97581061-e864-4439-95d2-458e444b18cc\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") "
Nov 01 22:45:46.975035 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.974925    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97581061-e864-4439-95d2-458e444b18cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97581061-e864-4439-95d2-458e444b18cc-maas-api-openshift-ca\") pod \"97581061-e864-4439-95d2-458e444b18cc\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") "
Nov 01 22:45:46.975145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.975127    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97581061-e864-4439-95d2-458e444b18cc-kube-api-access-p9srb\" (UniqueName: \"kubernetes.io/projected/97581061-e864-4439-95d2-458e444b18cc-kube-api-access-p9srb\") pod \"97581061-e864-4439-95d2-458e444b18cc\" (UID: \"97581061-e864-4439-95d2-458e444b18cc\") "
Nov 01 22:45:46.975313 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.975287    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97581061-e864-4439-95d2-458e444b18cc-maas-api-openshift-ca" pod "97581061-e864-4439-95d2-458e444b18cc" (UID: "97581061-e864-4439-95d2-458e444b18cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:46.975735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.975711    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97581061-e864-4439-95d2-458e444b18cc-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:46.977096 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.977071    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97581061-e864-4439-95d2-458e444b18cc-maas-api-xks-ca" pod "97581061-e864-4439-95d2-458e444b18cc" (UID: "97581061-e864-4439-95d2-458e444b18cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:46.977190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:46.977174    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97581061-e864-4439-95d2-458e444b18cc-kube-api-access-p9srb" pod "97581061-e864-4439-95d2-458e444b18cc" (UID: "97581061-e864-4439-95d2-458e444b18cc"). InnerVolumeSpecName "kube-api-access-p9srb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:47.080249 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.079074    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97581061-e864-4439-95d2-458e444b18cc-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:47.080249 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.079122    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9srb\" (UniqueName: \"kubernetes.io/projected/97581061-e864-4439-95d2-458e444b18cc-kube-api-access-p9srb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:47.512298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.512254    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:47.514307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.514280    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 01 22:45:47.514446 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.514331    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 01 22:45:47.712738 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.712709    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"
Nov 01 22:45:47.712928 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.712739    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht" event={"ID":"97581061-e864-4439-95d2-458e444b18cc","Type":"ContainerDied","Data":"1c899cb8ebc46da6fb06742a1cfd6eec54ea635842aa82c3c34e1ec2c6347652"}
Nov 01 22:45:47.712928 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.712790    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c899cb8ebc46da6fb06742a1cfd6eec54ea635842aa82c3c34e1ec2c6347652"
Nov 01 22:45:47.875530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:47.875446    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:48.512357 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.512319    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 01 22:45:48.512748 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.512374    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 01 22:45:48.681130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.681096    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"]
Nov 01 22:45:48.681839 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.681815    4837 cpu_manager.go:414] "containerMap: removing container" podUID="97581061-e864-4439-95d2-458e444b18cc" containerName="revoke-keys"
Nov 01 22:45:48.681839 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.681834    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97581061-e864-4439-95d2-458e444b18cc" containerName="revoke-keys"
Nov 01 22:45:48.681993 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.681962    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="97581061-e864-4439-95d2-458e444b18cc" containerName="revoke-keys"
Nov 01 22:45:48.685723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.685708    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.702155 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.702131    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplx7\" (UniqueName: \"kubernetes.io/projected/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-kube-api-access-pplx7\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.702261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.702190    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.702410 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.702392    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.720465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.720441    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:48.721426 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.721398    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:45:48.762141 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.762112    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"]
Nov 01 22:45:48.779814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.779745    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:45:48.779966 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.779949    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:48.804219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.804137    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.804603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.804559    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplx7\" (UniqueName: \"kubernetes.io/projected/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-kube-api-access-pplx7\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.806370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.804772    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.806926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.806906    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.808928 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.808890    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:48.822999 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:45:48.822971    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:45:48.863015 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:48.862990    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplx7\" (UniqueName: \"kubernetes.io/projected/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-kube-api-access-pplx7\") pod \"maas-api-revoke-sub-20ee0c0d-9dkkl\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:49.003229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:49.003192    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:49.464835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:49.464804    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"]
Nov 01 22:45:49.468643 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:49.468615    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a7b442_6fc7_4a6c_a1ba_d3a4c1a6a36f.slice/crio-5f6a28f1526cb3dd760d9bf5416e859ad8f6ba588d5bd04dfe3f696121c5c8e1 WatchSource:0}: Error finding container 5f6a28f1526cb3dd760d9bf5416e859ad8f6ba588d5bd04dfe3f696121c5c8e1: Status 404 returned error can't find the container with id 5f6a28f1526cb3dd760d9bf5416e859ad8f6ba588d5bd04dfe3f696121c5c8e1
Nov 01 22:45:49.512219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:49.512199    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 01 22:45:49.512292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:49.512242    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 01 22:45:49.729370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:49.729285    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl" event={"ID":"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f","Type":"ContainerStarted","Data":"5f6a28f1526cb3dd760d9bf5416e859ad8f6ba588d5bd04dfe3f696121c5c8e1"}
Nov 01 22:45:50.512348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:50.512321    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 01 22:45:50.512447 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:50.512382    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 01 22:45:50.758436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:50.758341    4837 generic.go:356] "Generic (PLEG): container finished" podID="58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f" containerID="f444ae0f3c20f38d20272cd3b8c4f58edf0624de7ef7e2c8163211010ece614b" exitCode=0
Nov 01 22:45:50.758834 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:50.758437    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl" event={"ID":"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f","Type":"ContainerDied","Data":"f444ae0f3c20f38d20272cd3b8c4f58edf0624de7ef7e2c8163211010ece614b"}
Nov 01 22:45:51.521360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.521322    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 01 22:45:51.521559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.521382    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 01 22:45:51.568844 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.568811    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"]
Nov 01 22:45:51.574106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.574081    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:51.586941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.586916    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f545de2-43d8-43cc-95ee-2ced75d4e05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9\" (UID: \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:51.587030 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.586960    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76tfh\" (UniqueName: \"kubernetes.io/projected/9f545de2-43d8-43cc-95ee-2ced75d4e05d-kube-api-access-76tfh\") pod \"payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9\" (UID: \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:51.639945 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.639902    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d5014777\""
Nov 01 22:45:51.660687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.660661    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"]
Nov 01 22:45:51.768757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.768712    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f545de2-43d8-43cc-95ee-2ced75d4e05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9\" (UID: \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:51.769577 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.769545    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76tfh\" (UniqueName: \"kubernetes.io/projected/9f545de2-43d8-43cc-95ee-2ced75d4e05d-kube-api-access-76tfh\") pod \"payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9\" (UID: \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:51.796128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:51.771002    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f545de2-43d8-43cc-95ee-2ced75d4e05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9\" (UID: \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:52.124146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.123906    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76tfh\" (UniqueName: \"kubernetes.io/projected/9f545de2-43d8-43cc-95ee-2ced75d4e05d-kube-api-access-76tfh\") pod \"payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9\" (UID: \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:52.189390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.189295    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:52.360567 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.360532    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:52.511846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.511807    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-xks-ca\") pod \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") "
Nov 01 22:45:52.512027 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.511979    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-kube-api-access-pplx7\" (UniqueName: \"kubernetes.io/projected/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-kube-api-access-pplx7\") pod \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") "
Nov 01 22:45:52.512142 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.512123    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-openshift-ca\") pod \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\" (UID: \"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f\") "
Nov 01 22:45:52.512315 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.512280    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 01 22:45:52.512433 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.512368    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 01 22:45:52.512551 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.512524    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-openshift-ca" pod "58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f" (UID: "58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:52.512867 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.512844    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:52.514440 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.514420    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-kube-api-access-pplx7" pod "58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f" (UID: "58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f"). InnerVolumeSpecName "kube-api-access-pplx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:52.514516 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.514422    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-xks-ca" pod "58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f" (UID: "58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:52.615029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.615000    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:52.615204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.615186    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplx7\" (UniqueName: \"kubernetes.io/projected/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f-kube-api-access-pplx7\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:52.615281 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.615247    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"]
Nov 01 22:45:52.906493 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.906453    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"]
Nov 01 22:45:52.907896 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.907410    4837 cpu_manager.go:414] "containerMap: removing container" podUID="58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f" containerName="revoke-keys"
Nov 01 22:45:52.907896 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.907431    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f" containerName="revoke-keys"
Nov 01 22:45:52.907896 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.907618    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f" containerName="revoke-keys"
Nov 01 22:45:52.912221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.912201    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.011852 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:52.997222    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b50b3403-openshift-default-dockercfg-zksxb\""
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.039084    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-envoy\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.039156    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-token\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.068176    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-certs\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.068351    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7xcz\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-kube-api-access-d7xcz\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.068424    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23ce78c-1db4-46e2-b04c-31c56f139dca-istiod-ca-cert\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.068503    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-socket\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.068560    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-data\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.068607    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-podinfo\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.103119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.068668    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-credential-socket\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.189334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.180135    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-certs\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.189334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.180343    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7xcz\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-kube-api-access-d7xcz\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.189334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.180427    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23ce78c-1db4-46e2-b04c-31c56f139dca-istiod-ca-cert\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.189334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.180504    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-socket\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.195071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.191446    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-certs\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.200021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.199887    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23ce78c-1db4-46e2-b04c-31c56f139dca-istiod-ca-cert\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.238554 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.233948    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-data\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.238554 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.234027    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-podinfo\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.238554 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.234111    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-credential-socket\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.271078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.266496    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-data\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.273357 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.273325    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-socket\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.275873 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.275844    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-credential-socket\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.279204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.279175    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-envoy\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.279407 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.279389    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-token\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.345598 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.320270    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-envoy\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.357102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.356766    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"]
Nov 01 22:45:53.392846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.391343    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl" event={"ID":"58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f","Type":"ContainerDied","Data":"5f6a28f1526cb3dd760d9bf5416e859ad8f6ba588d5bd04dfe3f696121c5c8e1"}
Nov 01 22:45:53.393135 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.393111    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f6a28f1526cb3dd760d9bf5416e859ad8f6ba588d5bd04dfe3f696121c5c8e1"
Nov 01 22:45:53.393357 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.393343    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"
Nov 01 22:45:53.458072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.458019    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-podinfo\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.462992 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.462958    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9" event={"ID":"9f545de2-43d8-43cc-95ee-2ced75d4e05d","Type":"ContainerStarted","Data":"a1b4211aff94ba8745888b109cf3e337aa300e21a03a45e6125d19d334958de8"}
Nov 01 22:45:53.463254 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.463233    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9" event={"ID":"9f545de2-43d8-43cc-95ee-2ced75d4e05d","Type":"ContainerStarted","Data":"546fa9f9f33fd4d7c9edee849de46db89ce4d7f6ecad78b47c198a0e36fed3b8"}
Nov 01 22:45:53.464267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.464248    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:45:53.472815 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.472786    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7xcz\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-kube-api-access-d7xcz\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.513342 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.513305    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 01 22:45:53.513889 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.513856    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 01 22:45:53.535620 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.528959    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-token\") pod \"e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") " pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:53.663117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.660718    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9" podStartSLOduration=2.6606988940000003 podStartE2EDuration="2.660698894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:53.644567878 +0000 UTC m=+1335.233766965" watchObservedRunningTime="2026-11-01 22:45:53.660698894 +0000 UTC m=+1335.249897956"
Nov 01 22:45:53.733333 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:53.733303    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:54.239854 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.239779    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"]
Nov 01 22:45:54.255572 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:54.255544    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23ce78c_1db4_46e2_b04c_31c56f139dca.slice/crio-131112af802209d807e3fff2090660b8c3db551394cb89836618181c4334629e WatchSource:0}: Error finding container 131112af802209d807e3fff2090660b8c3db551394cb89836618181c4334629e: Status 404 returned error can't find the container with id 131112af802209d807e3fff2090660b8c3db551394cb89836618181c4334629e
Nov 01 22:45:54.258477 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.258448    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:54.258552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.258518    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:54.258601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.258551    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:54.477483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.477438    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" event={"ID":"e23ce78c-1db4-46e2-b04c-31c56f139dca","Type":"ContainerStarted","Data":"077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f"}
Nov 01 22:45:54.477649 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.477492    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" event={"ID":"e23ce78c-1db4-46e2-b04c-31c56f139dca","Type":"ContainerStarted","Data":"131112af802209d807e3fff2090660b8c3db551394cb89836618181c4334629e"}
Nov 01 22:45:54.516403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.516336    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:54.516626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.516606    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:54.525607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.525583    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:45:54.605102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.605010    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podStartSLOduration=2.6049962239999997 podStartE2EDuration="2.604996224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:54.602559901 +0000 UTC m=+1336.191758961" watchObservedRunningTime="2026-11-01 22:45:54.604996224 +0000 UTC m=+1336.194195287"
Nov 01 22:45:54.734430 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:54.734119    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:55.739986 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:55.739957    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:56.492970 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:56.492936    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:56.494215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:56.494193    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:45:57.002298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.002267    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"]
Nov 01 22:45:57.006690 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.006670    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.099961 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.099921    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.116429 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.116398    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.116784 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.116763    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.116976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.116961    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a66841d-00fa-47c5-a555-573984719a34-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.117211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.117192    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.117387 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.117371    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8k6v\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-kube-api-access-w8k6v\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.117671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.117655    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a66841d-00fa-47c5-a555-573984719a34-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.117814 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.117800    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.117949 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.117936    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.135096 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.131169    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"]
Nov 01 22:45:57.219510 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.219471    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a66841d-00fa-47c5-a555-573984719a34-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.219817 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.219775    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.219967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.219948    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8k6v\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-kube-api-access-w8k6v\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.220231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.220212    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a66841d-00fa-47c5-a555-573984719a34-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.220347 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.220334    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.220469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.220456    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.220594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.220581    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.220773 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.220758    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.220933 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.220919    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.223933 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.223907    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a66841d-00fa-47c5-a555-573984719a34-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.233199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.231488    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.233199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.231737    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.233199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.231956    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.233757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.233737    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.248284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.247374    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.301089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.290959    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a66841d-00fa-47c5-a555-573984719a34-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.319085 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.319031    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.440073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.424794    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8k6v\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-kube-api-access-w8k6v\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:57.620464 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:57.620377    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:58.130903 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:45:58.130876    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a66841d_00fa_47c5_a555_573984719a34.slice/crio-6104327f17df23de1bb8c03afb1b5862853e7165a6f51077432a4d25dc476996 WatchSource:0}: Error finding container 6104327f17df23de1bb8c03afb1b5862853e7165a6f51077432a4d25dc476996: Status 404 returned error can't find the container with id 6104327f17df23de1bb8c03afb1b5862853e7165a6f51077432a4d25dc476996
Nov 01 22:45:58.131671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:58.131643    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"]
Nov 01 22:45:58.133716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:58.133680    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:58.133800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:58.133749    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:58.133800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:58.133782    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:45:58.530556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:58.530523    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" event={"ID":"0a66841d-00fa-47c5-a555-573984719a34","Type":"ContainerStarted","Data":"5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72"}
Nov 01 22:45:58.530747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:58.530571    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" event={"ID":"0a66841d-00fa-47c5-a555-573984719a34","Type":"ContainerStarted","Data":"6104327f17df23de1bb8c03afb1b5862853e7165a6f51077432a4d25dc476996"}
Nov 01 22:45:58.620707 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:58.620673    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:45:59.627072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:59.625946    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:45:59.627072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:45:59.626024    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:00.621744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:00.621715    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:00.621910 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:00.621760    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:01.621207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:01.621169    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:01.621586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:01.621235    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:02.724068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:02.668513    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:02.761869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:02.758165    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:03.511076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.502796    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podStartSLOduration=7.502721314 podStartE2EDuration="7.502721314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:58.746077048 +0000 UTC m=+1340.335276106" watchObservedRunningTime="2026-11-01 22:46:03.502721314 +0000 UTC m=+1345.091920375"
Nov 01 22:46:03.529602 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.529561    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"]
Nov 01 22:46:03.565502 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.565458    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.570068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.566590    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.595520 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.595475    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:03.633637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.632560    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/1d631fdf-76e9-46b4-afd2-81104f146f07-kube-api-access-2nbdd\") pod \"payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf\" (UID: \"1d631fdf-76e9-46b4-afd2-81104f146f07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:03.633637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.632641    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d631fdf-76e9-46b4-afd2-81104f146f07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf\" (UID: \"1d631fdf-76e9-46b4-afd2-81104f146f07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:03.648074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.645804    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:03.648074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.645862    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:03.648074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.646687    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"]
Nov 01 22:46:03.652086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.649871    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b50b3403\""
Nov 01 22:46:03.687458 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.687428    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b50b3403-dockercfg-lbqrc\""
Nov 01 22:46:03.772359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.772267    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:46:03.795627 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.772995    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:46:03.806576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.774684    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/1d631fdf-76e9-46b4-afd2-81104f146f07-kube-api-access-2nbdd\") pod \"payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf\" (UID: \"1d631fdf-76e9-46b4-afd2-81104f146f07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:03.806576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.798021    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d631fdf-76e9-46b4-afd2-81104f146f07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf\" (UID: \"1d631fdf-76e9-46b4-afd2-81104f146f07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:03.806576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.799281    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d631fdf-76e9-46b4-afd2-81104f146f07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf\" (UID: \"1d631fdf-76e9-46b4-afd2-81104f146f07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:03.828070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.810359    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.902066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:03.881090    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/1d631fdf-76e9-46b4-afd2-81104f146f07-kube-api-access-2nbdd\") pod \"payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf\" (UID: \"1d631fdf-76e9-46b4-afd2-81104f146f07\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:04.042556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.040104    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.042556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.042334    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:04.129498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.129331    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 01 22:46:04.256530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.256217    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.332099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.331995    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.372179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.372137    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.373283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.373253    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:04.373600 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.373545    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.384169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.384129    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.571412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.571259    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.571412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.571355    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.630714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.630623    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:04.630978 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.630939    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:04.739714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.739680    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:46:04.740118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.740100    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:46:04.804194 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.804147    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.823664 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.805079    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.824192 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.806522    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.824553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.812113    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.824866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.823898    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:46:04.825021 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.825008    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:46:04.826087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:04.826070    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:46:05.058289 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.058246    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.099206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.099175    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:46:05.247066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.207949    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.313146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.272981    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:46:05.326070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.313717    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:46:05.387083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.342195    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.519088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.393484    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.528071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.521346    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:46:05.528590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.528564    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:46:05.536083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.393531    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.550298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.539162    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:46:05.985114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.845175    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.108332 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.845655    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.108643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.845693    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.116148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:05.849136    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.196570 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.196513    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.225212 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.196984    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.257560 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.257468    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:06.257853 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.257820    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.287272 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.287233    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:46:06.339465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.339416    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.406383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.406338    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:06.443090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.443030    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.520191 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.518807    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.648893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.616842    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:06.648893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.616932    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:06.648893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.617341    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.648893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.617425    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.773744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.773647    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.826979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.791289    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:06.948129 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.946857    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:06.948129 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.847880    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq565" podUID="87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.959002 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:06.949376    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4jbsp" podUID="7ea44462-9d6a-42ec-ae71-4f8b3567f7b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.090727 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.090628    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.198249 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.198205    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.266976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.231506    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.283180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.282362    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.395058 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.348450    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.395614 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.348502    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.664187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.649373    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:07.664187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.649435    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:07.700087 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:46:07.699336    4837 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.181s"
Nov 01 22:46:07.700087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.699424    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:46:07.700087 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.699508    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"]
Nov 01 22:46:07.733200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.733154    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.733753 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.733720    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.780913 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.780870    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.781540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.781155    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.830000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.829951    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.915199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.898146    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Nov 01 22:46:07.915199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.912738    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Nov 01 22:46:07.951474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:07.951428    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.119069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.117276    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.163219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.163173    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf" event={"ID":"1d631fdf-76e9-46b4-afd2-81104f146f07","Type":"ContainerStarted","Data":"058ec4d10b4b522969184cc3302ae9810a16801d04ceea357f4246828fc23225"}
Nov 01 22:46:08.279586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.269137    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.287078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.277239    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.405067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.396151    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.405067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.396558    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.405067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.396865    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.423524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.423493    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:46:08.636671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.636578    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:08.636946 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.636913    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.638253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.638230    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:08.638405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.638381    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:08.753296 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.753255    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.753635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.753602    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.783119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.782734    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.932132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.907070    4837 trace.go:236] Trace[652439537]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hmhc4" (01-Nov-2026 22:46:07.844) (total time: 1062ms):
Nov 01 22:46:08.932132 ip-10-0-129-77 kubenswrapper[4837]: Trace[652439537]: [1.062631944s] [1.062631944s] END
Nov 01 22:46:09.086553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.974703    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:09.086553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.079294    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.086553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:08.995699    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.106354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.095307    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.132565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.132284    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.285335 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.271582    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.285335 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.285094    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.349315 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.348997    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:46:09.402275 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.402235    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.404184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.404157    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.438502 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.430487    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.630176 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.627159    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:09.630176 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.627229    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:09.759084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.756801    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.759084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.758547    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.784065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:09.782895    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.057235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.033020    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 22:46:10.191707 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.154156    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.274212 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.274117    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.290182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.290141    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.307018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.299966    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:46:10.397156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.397005    4837 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:10.397156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.397080    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.408839 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.407465    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.408839 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.407768    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.474093 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.461080    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.474093 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.461706    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:46:10.787270 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.787225    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.877096 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.875205    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.952837 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.952797    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.968112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.968082    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:10.968375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:10.968353    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:11.057174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.044885    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:46:11.060025 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.057664    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:46:11.161642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.160876    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.165780 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.165752    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:46:11.175125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.175096    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:46:11.210179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.201979    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:46:11.210179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.202104    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:46:11.271119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.268845    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:46:11.288473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.288440    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:46:11.621609 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.621575    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:46:11.621804 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.621631    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:46:11.751579 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.751545    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"]
Nov 01 22:46:11.757127 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.757107    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:11.825104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.824197    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:46:11.825104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.824373    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:46:11.909764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.909666    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"]
Nov 01 22:46:11.930972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.930936    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:11.931249 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.931228    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v67z8\" (UniqueName: \"kubernetes.io/projected/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-kube-api-access-v67z8\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:11.931563 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:11.931545    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:12.035618 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.035573    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:12.036155 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.036117    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:12.036332 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.036313    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v67z8\" (UniqueName: \"kubernetes.io/projected/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-kube-api-access-v67z8\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:12.036548 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.036523    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:12.042445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.042419    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:12.094590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.094510    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v67z8\" (UniqueName: \"kubernetes.io/projected/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-kube-api-access-v67z8\") pod \"maas-api-revoke-sub-b4784835-2m24n\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:12.230975 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.230937    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf" event={"ID":"1d631fdf-76e9-46b4-afd2-81104f146f07","Type":"ContainerStarted","Data":"51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd"}
Nov 01 22:46:12.231504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.231179    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:12.355135 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.355069    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf" podStartSLOduration=9.355033745 podStartE2EDuration="9.355033745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:12.354116991 +0000 UTC m=+1353.943316052" watchObservedRunningTime="2026-11-01 22:46:12.355033745 +0000 UTC m=+1353.944232807"
Nov 01 22:46:12.390568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.390534    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:12.621401 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.621319    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:46:12.791407 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:12.791379    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"]
Nov 01 22:46:13.245355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:13.245313    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n" event={"ID":"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6","Type":"ContainerStarted","Data":"adc9e63c6c278416eb8489bc6161dfddc0b4984ed91ad6f337a0ab192a63f25a"}
Nov 01 22:46:13.245866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:13.245382    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n" event={"ID":"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6","Type":"ContainerStarted","Data":"b39d4d2942bc70918db961879094cb78aa4a0c048b6f01134a0b43b6392fd961"}
Nov 01 22:46:13.357148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:13.357038    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n" podStartSLOduration=2.357024353 podStartE2EDuration="2.357024353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:13.354299471 +0000 UTC m=+1354.943498548" watchObservedRunningTime="2026-11-01 22:46:13.357024353 +0000 UTC m=+1354.946223448"
Nov 01 22:46:13.716067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:13.705178    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:13.716067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:13.705252    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:14.062383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:14.062352    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:46:14.064077 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:14.064034    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:46:14.254221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:14.254186    4837 generic.go:356] "Generic (PLEG): container finished" podID="9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6" containerID="adc9e63c6c278416eb8489bc6161dfddc0b4984ed91ad6f337a0ab192a63f25a" exitCode=0
Nov 01 22:46:14.254590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:14.254234    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n" event={"ID":"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6","Type":"ContainerDied","Data":"adc9e63c6c278416eb8489bc6161dfddc0b4984ed91ad6f337a0ab192a63f25a"}
Nov 01 22:46:15.484858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.484821    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:15.655194 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.655116    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-xks-ca\") pod \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") "
Nov 01 22:46:15.655354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.655260    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-kube-api-access-v67z8\" (UniqueName: \"kubernetes.io/projected/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-kube-api-access-v67z8\") pod \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") "
Nov 01 22:46:15.655424 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.655374    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-openshift-ca\") pod \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\" (UID: \"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6\") "
Nov 01 22:46:15.655800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.655764    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-openshift-ca" pod "9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6" (UID: "9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:46:15.656371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.656342    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:15.657482 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.657459    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-kube-api-access-v67z8" pod "9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6" (UID: "9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6"). InnerVolumeSpecName "kube-api-access-v67z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:46:15.657482 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.657468    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-xks-ca" pod "9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6" (UID: "9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:46:15.757996 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.757962    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:15.757996 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:15.757990    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v67z8\" (UniqueName: \"kubernetes.io/projected/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6-kube-api-access-v67z8\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:16.273285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:16.273253    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"
Nov 01 22:46:16.273285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:16.273264    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n" event={"ID":"9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6","Type":"ContainerDied","Data":"b39d4d2942bc70918db961879094cb78aa4a0c048b6f01134a0b43b6392fd961"}
Nov 01 22:46:16.273524 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:16.273321    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b39d4d2942bc70918db961879094cb78aa4a0c048b6f01134a0b43b6392fd961"
Nov 01 22:46:18.813335 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:46:18.813302    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:46:23.248985 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:23.248956    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:46:27.317912 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.317878    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"]
Nov 01 22:46:27.318759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.318737    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6" containerName="revoke-keys"
Nov 01 22:46:27.318759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.318758    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6" containerName="revoke-keys"
Nov 01 22:46:27.318979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.318965    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6" containerName="revoke-keys"
Nov 01 22:46:27.322826 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.322804    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.364092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.364067    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"]
Nov 01 22:46:27.368796 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.368779    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.381360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.381337    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bv4kj\""
Nov 01 22:46:27.411600 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.411569    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"]
Nov 01 22:46:27.436674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.436641    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.436858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.436708    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.436858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.436810    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.436979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.436889    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.436979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.436954    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k88\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-kube-api-access-v4k88\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.437145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.436993    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.437145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437022    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.437145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437113    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.437145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437138    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.437340 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437205    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.437340 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437240    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.437423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437358    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.437423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437384    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.437423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437403    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xndr\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-kube-api-access-6xndr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.437591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437572    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.437686 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437667    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.437726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437706    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.437769 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.437756    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828731a6-88ec-40b3-ac92-e57431e8dab8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.462312 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.462276    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"]
Nov 01 22:46:27.539287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539258    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.539287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539293    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.539534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539339    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828731a6-88ec-40b3-ac92-e57431e8dab8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.539534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539411    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.539534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539492    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.539677 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539579    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.539677 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539656    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.539771 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539709    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k88\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-kube-api-access-v4k88\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.539771 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539744    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.539874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539767    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.539874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539812    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.539874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539819    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.539874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539846    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.540125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539878    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.540125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539910    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.540125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.539959    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.540125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540003    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.540125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540020    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.540391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540204    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828731a6-88ec-40b3-ac92-e57431e8dab8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.540391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540028    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.540391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540274    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xndr\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-kube-api-access-6xndr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.540391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540312    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.540578 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540386    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.540578 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540414    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.540578 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540519    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.540802 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540776    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.540914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540798    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.540914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.540815    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.542265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.542242    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.542644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.542619    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.542849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.542831    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.542908 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.542857    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.594612 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.594548    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.594612 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.594569    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k88\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-kube-api-access-v4k88\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.594782 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.594758    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hdrlk\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:27.594851 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.594835    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xndr\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-kube-api-access-6xndr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hgg9r\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.636964 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.636940    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:27.683013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:27.682982    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:28.059248 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:46:28.059215    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828731a6_88ec_40b3_ac92_e57431e8dab8.slice/crio-6ef617ce77400ace34f4038c553b1b51e3f15d4b2ec8d5d0d18f0b3d3d1ef1e1 WatchSource:0}: Error finding container 6ef617ce77400ace34f4038c553b1b51e3f15d4b2ec8d5d0d18f0b3d3d1ef1e1: Status 404 returned error can't find the container with id 6ef617ce77400ace34f4038c553b1b51e3f15d4b2ec8d5d0d18f0b3d3d1ef1e1
Nov 01 22:46:28.059248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.059237    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"]
Nov 01 22:46:28.062115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.062084    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:28.062203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.062154    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:28.062203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.062181    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:28.106770 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:46:28.106743    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4835c74_5ce8_4b54_8c88_49aa8b706df0.slice/crio-abe362a3128b484f3beae0d4e8f2f3444238d1e18f6dec6cb9de4c8f035a3fa1 WatchSource:0}: Error finding container abe362a3128b484f3beae0d4e8f2f3444238d1e18f6dec6cb9de4c8f035a3fa1: Status 404 returned error can't find the container with id abe362a3128b484f3beae0d4e8f2f3444238d1e18f6dec6cb9de4c8f035a3fa1
Nov 01 22:46:28.108527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.108508    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"]
Nov 01 22:46:28.109612 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.109584    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:28.109678 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.109646    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:28.109715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.109677    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:28.368705 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.368606    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" event={"ID":"e4835c74-5ce8-4b54-8c88-49aa8b706df0","Type":"ContainerStarted","Data":"6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81"}
Nov 01 22:46:28.369244 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.368706    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" event={"ID":"e4835c74-5ce8-4b54-8c88-49aa8b706df0","Type":"ContainerStarted","Data":"abe362a3128b484f3beae0d4e8f2f3444238d1e18f6dec6cb9de4c8f035a3fa1"}
Nov 01 22:46:28.370633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.370603    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" event={"ID":"828731a6-88ec-40b3-ac92-e57431e8dab8","Type":"ContainerStarted","Data":"f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3"}
Nov 01 22:46:28.370767 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.370649    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" event={"ID":"828731a6-88ec-40b3-ac92-e57431e8dab8","Type":"ContainerStarted","Data":"6ef617ce77400ace34f4038c553b1b51e3f15d4b2ec8d5d0d18f0b3d3d1ef1e1"}
Nov 01 22:46:28.482778 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.482722    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podStartSLOduration=1.482707702 podStartE2EDuration="1.482707702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:28.481205968 +0000 UTC m=+1370.070405042" watchObservedRunningTime="2026-11-01 22:46:28.482707702 +0000 UTC m=+1370.071906762"
Nov 01 22:46:28.598296 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.598231    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podStartSLOduration=1.598217157 podStartE2EDuration="1.598217157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:28.595029786 +0000 UTC m=+1370.184228848" watchObservedRunningTime="2026-11-01 22:46:28.598217157 +0000 UTC m=+1370.187416218"
Nov 01 22:46:28.639776 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.639678    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:28.684299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:28.684263    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:29.641070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:29.639865    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 22:46:29.641070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:29.639938    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 22:46:29.684943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:29.684641    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hdrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:46:29.684943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:29.684691    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:46:30.652066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:30.648386    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 22:46:30.652066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:30.648458    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 22:46:30.683642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:30.683589    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hdrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:46:30.683826 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:30.683662    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:46:31.670265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:31.668632    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 22:46:31.670265 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:31.668691    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 22:46:31.683521 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:31.683490    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hdrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:46:31.683801 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:31.683774    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:46:32.649068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:32.641390    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 22:46:32.649068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:32.641463    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 22:46:32.683490 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:32.683458    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hdrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:46:32.683860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:32.683512    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:46:33.649064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.646135    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 22:46:33.649064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.646195    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 22:46:33.684118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.683413    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hdrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:46:33.684118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.683472    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:46:33.686643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.686616    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"]
Nov 01 22:46:33.695136 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.695111    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.726948 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.726914    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.727094 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.726991    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-metrics-certs\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.727175 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.727140    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtsh\" (UniqueName: \"kubernetes.io/projected/0e34436a-4e94-4692-85ad-7cd548e952ad-kube-api-access-kbtsh\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.727244 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.727201    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34436a-4e94-4692-85ad-7cd548e952ad-openshift-service-ca\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.760295 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.760263    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d5014777-serving-cert\""
Nov 01 22:46:33.792349 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.792315    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"]
Nov 01 22:46:33.829173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.829138    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtsh\" (UniqueName: \"kubernetes.io/projected/0e34436a-4e94-4692-85ad-7cd548e952ad-kube-api-access-kbtsh\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.829343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.829233    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34436a-4e94-4692-85ad-7cd548e952ad-openshift-service-ca\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.829496 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.829474    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.829699 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:46:33.829675    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-serving-cert: secret "maas-api-e2e-ait-d5014777-serving-cert" not found
Nov 01 22:46:33.829766 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:46:33.829753    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls podName:0e34436a-4e94-4692-85ad-7cd548e952ad nodeName:}" failed. No retries permitted until 2026-11-01 22:46:34.329732366 +0000 UTC m=+1375.918931422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls") pod "maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7" (UID: "0e34436a-4e94-4692-85ad-7cd548e952ad") : secret "maas-api-e2e-ait-d5014777-serving-cert" not found
Nov 01 22:46:33.829901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.829885    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-metrics-certs\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.830295 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.830269    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34436a-4e94-4692-85ad-7cd548e952ad-openshift-service-ca\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.882068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.881447    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-metrics-certs\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:33.896055 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:33.896009    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtsh\" (UniqueName: \"kubernetes.io/projected/0e34436a-4e94-4692-85ad-7cd548e952ad-kube-api-access-kbtsh\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:34.360715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:34.360676    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:34.418380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:34.418350    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls\") pod \"maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:34.617943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:34.617842    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:34.638118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:34.638090    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 22:46:34.638246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:34.638158    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 22:46:34.683547 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:34.683319    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hdrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:46:34.683547 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:34.683371    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:46:35.041945 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:35.041913    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"]
Nov 01 22:46:35.045277 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:46:35.045247    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e34436a_4e94_4692_85ad_7cd548e952ad.slice/crio-00efa7a6f5f266482da7523252aa73e95b8878df68d418c80993c3e68fcaf79a WatchSource:0}: Error finding container 00efa7a6f5f266482da7523252aa73e95b8878df68d418c80993c3e68fcaf79a: Status 404 returned error can't find the container with id 00efa7a6f5f266482da7523252aa73e95b8878df68d418c80993c3e68fcaf79a
Nov 01 22:46:35.046765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:35.046745    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:46:35.452309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:35.452272    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7" event={"ID":"0e34436a-4e94-4692-85ad-7cd548e952ad","Type":"ContainerStarted","Data":"00efa7a6f5f266482da7523252aa73e95b8878df68d418c80993c3e68fcaf79a"}
Nov 01 22:46:35.637671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:35.637640    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 22:46:35.637842 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:35.637687    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 22:46:35.683675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:35.683644    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hdrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:46:35.683854 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:35.683694    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:46:36.467119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:36.467079    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7" event={"ID":"0e34436a-4e94-4692-85ad-7cd548e952ad","Type":"ContainerStarted","Data":"2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481"}
Nov 01 22:46:36.467633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:36.467158    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:36.583892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:36.583823    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7" podStartSLOduration=2.7215449830000003 podStartE2EDuration="3.583804326s" totalImagesPullingTime="862.259343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:36.580395745 +0000 UTC m=+1378.169594807" watchObservedRunningTime="2026-11-01 22:46:36.583804326 +0000 UTC m=+1378.173003388"
Nov 01 22:46:37.629433 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:37.629399    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:37.633203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:37.633179    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:37.634400 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:37.634375    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 22:46:37.638782 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:37.638760    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:37.638942 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:37.638921    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:37.648726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:37.647448    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:37.648726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:37.648565    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:37.649508 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:37.649489    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 22:46:38.750307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:38.750275    4837 scope.go:122] "RemoveContainer" containerID="826b9629a7e1b4f0c1cc98a07ef558e6c93973aaedb2604734df269d667a2ca4"
Nov 01 22:46:38.763318 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:38.763296    4837 scope.go:122] "RemoveContainer" containerID="a0f9001328d94399ce1ed3a677ebc318874c16002d76f9836334d98920fe3c57"
Nov 01 22:46:38.771971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:38.771956    4837 scope.go:122] "RemoveContainer" containerID="1414211422dc9f0bdcae384f5220f26b2ebb8dd25592ba52cfe1f0f7c1db7a46"
Nov 01 22:46:38.780682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:38.780667    4837 scope.go:122] "RemoveContainer" containerID="172b16c90ef6164a532d00ae357919cd51a0af4b2a7b0f0def7eb0e5fd0ca76b"
Nov 01 22:46:38.788948 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:38.788932    4837 scope.go:122] "RemoveContainer" containerID="9548bf760663736afaa6a323f81bd9230604868fb1461a0e12a9c19851f4784a"
Nov 01 22:46:41.821107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:41.821074    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:46:42.465679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.463361    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"]
Nov 01 22:46:42.492217 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.487174    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.503011 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.502983    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:46:42.521568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.521536    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"]
Nov 01 22:46:42.537479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.537453    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.590345 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.590313    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ac7195-82b7-4b1a-99de-21e23225824b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.590575 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.590560    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.590712 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.590699    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.590811 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.590800    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.590953 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.590938    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.591111 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.591034    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629d787b-d1d4-4102-90dc-2c095dd59a56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.591253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.591234    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.591427 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.591413    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.591543 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.591531    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.591676 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.591661    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ac7195-82b7-4b1a-99de-21e23225824b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.591884 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.591870    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.592000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.591987    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.592162 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.592146    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqlhk\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-kube-api-access-qqlhk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.592304 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.592290    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.592466 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.592453    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.592630 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.592615    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.592732 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.592719    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.592843 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.592829    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5px\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-kube-api-access-8t5px\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.593364 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.592649    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"]
Nov 01 22:46:42.630245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.630204    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"]
Nov 01 22:46:42.694451 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.694419    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ac7195-82b7-4b1a-99de-21e23225824b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.694619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.694510    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.694619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.694550    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.694619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.694598    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqlhk\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-kube-api-access-qqlhk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.694904 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.694881    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.695017 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.694992    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.695188 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695096    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.695331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695096    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.695331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695287    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.695331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695318    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.695545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695350    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5px\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-kube-api-access-8t5px\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.695545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695369    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.695545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695422    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ac7195-82b7-4b1a-99de-21e23225824b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.695545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695455    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.695545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695517    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.695865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695546    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.695865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695613    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.695865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695639    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629d787b-d1d4-4102-90dc-2c095dd59a56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.695865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695675    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.696144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.695956    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.696300 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.696240    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ac7195-82b7-4b1a-99de-21e23225824b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.696686 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.696472    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.696686 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.696588    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.697168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.696646    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.697168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.696842    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.697168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.697129    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.697474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.697451    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629d787b-d1d4-4102-90dc-2c095dd59a56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.697620 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.697536    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.699347 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.699298    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.700284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.700262    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.700657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.700616    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ac7195-82b7-4b1a-99de-21e23225824b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.701572 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.701551    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.760995 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.760833    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5px\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-kube-api-access-8t5px\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.763881 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.763830    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4bhlj\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.764652 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.764627    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqlhk\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-kube-api-access-qqlhk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.772706 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.767481    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pjdqk\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:42.901513 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.894340    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:42.917086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:42.913164    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:44.179114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.179082    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"]
Nov 01 22:46:44.227984 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.226896    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.235565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.233150    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.272066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.271170    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.272066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.271488    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.292286 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.289109    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.494076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.460174    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:44.494076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.460397    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:46:44.737236 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.737118    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.803185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:44.752682    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.010861 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.004022    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 22:46:45.051965 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:46:45.051767    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629d787b_d1d4_4102_90dc_2c095dd59a56.slice/crio-8cbe377aaeede55713bf20ee1c13033c2eba9526c0448389b94ac2352eb59dc6 WatchSource:0}: Error finding container 8cbe377aaeede55713bf20ee1c13033c2eba9526c0448389b94ac2352eb59dc6: Status 404 returned error can't find the container with id 8cbe377aaeede55713bf20ee1c13033c2eba9526c0448389b94ac2352eb59dc6
Nov 01 22:46:45.156758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.156714    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:45.158214 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.158190    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:45.158441 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.158421    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:45.203149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.201902    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:45.203149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.201971    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.239478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.239413    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:46:45.240843 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.240811    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.254311 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.253585    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.291464 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.291416    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.318807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.307842    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:45.318807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.308244    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:45.318807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.308484    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:46:45.329972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.328209    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:46:45.349809 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.349785    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-798f4550-serving-cert\""
Nov 01 22:46:45.355715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.355691    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-metrics-certs\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.356152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.356131    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mxq\" (UniqueName: \"kubernetes.io/projected/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-kube-api-access-v6mxq\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.356413 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.356394    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-maas-api-tls\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.356564 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.356550    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-openshift-service-ca\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.494423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.494393    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:46:45.494563 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.494457    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:46:45.494563 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.494478    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"]
Nov 01 22:46:45.494688 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.494574    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:46:45.494688 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.494592    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"]
Nov 01 22:46:45.494688 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.494623    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"]
Nov 01 22:46:45.500745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.497282    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:46:45.510821 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.510477    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 22:46:45.536475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.520541    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mxq\" (UniqueName: \"kubernetes.io/projected/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-kube-api-access-v6mxq\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.536475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.520643    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-maas-api-tls\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.536475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.520675    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-openshift-service-ca\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.536475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.522282    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-metrics-certs\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.536475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.524411    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:46:45.536475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.526834    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-openshift-service-ca\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.568073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.564024    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-maas-api-tls\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.635143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.629669    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-metrics-certs\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.684267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.674845    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mxq\" (UniqueName: \"kubernetes.io/projected/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-kube-api-access-v6mxq\") pod \"maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:45.982428 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:45.982343    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:46.006071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:46.005029    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:46.091226 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:46.050361    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:46:46.390027 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:46.389998    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:46:46.474097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:46.467654    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" event={"ID":"19ac7195-82b7-4b1a-99de-21e23225824b","Type":"ContainerStarted","Data":"8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1"}
Nov 01 22:46:46.474097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:46.467741    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" event={"ID":"19ac7195-82b7-4b1a-99de-21e23225824b","Type":"ContainerStarted","Data":"1220b04bc7f7448233e5820fcedf143555e40400635cd871371f2020fc535aeb"}
Nov 01 22:46:46.905334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:46.905255    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:46.905636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:46.905605    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:47.043077 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.039167    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podStartSLOduration=5.039141172 podStartE2EDuration="5.039141172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:46.890442389 +0000 UTC m=+1388.479641476" watchObservedRunningTime="2026-11-01 22:46:47.039141172 +0000 UTC m=+1388.628340240"
Nov 01 22:46:47.134070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.133140    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podStartSLOduration=5.133119577 podStartE2EDuration="5.133119577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:47.090560259 +0000 UTC m=+1388.679759318" watchObservedRunningTime="2026-11-01 22:46:47.133119577 +0000 UTC m=+1388.722318641"
Nov 01 22:46:47.166124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.136079    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" event={"ID":"629d787b-d1d4-4102-90dc-2c095dd59a56","Type":"ContainerStarted","Data":"db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7"}
Nov 01 22:46:47.166124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.136165    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" event={"ID":"629d787b-d1d4-4102-90dc-2c095dd59a56","Type":"ContainerStarted","Data":"8cbe377aaeede55713bf20ee1c13033c2eba9526c0448389b94ac2352eb59dc6"}
Nov 01 22:46:47.166124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.136199    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:47.166124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.136308    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:47.467411 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.467377    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"]
Nov 01 22:46:47.904264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.897537    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:47.904264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.897596    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:48.002086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.952356    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 22:46:48.002086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:47.952417    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 22:46:48.036152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.007891    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg" event={"ID":"83416f6e-dd38-4edc-80ca-6b6fa35e64ed","Type":"ContainerStarted","Data":"9d0566d55538c1ede0cd99a35821dff359ebe69b1c895a164a30b32fdaca3b49"}
Nov 01 22:46:48.262191 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.262143    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:48.319071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.311820    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:46:48.323712 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.319147    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:48.390106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.387161    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:48.456318 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.453954    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:48.497230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.497192    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:48.580878 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.580773    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:48.924121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.905495    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:48.924121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.905583    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:48.998195 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.998157    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 22:46:48.998472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:48.998437    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 22:46:49.015345 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:46:49.005564    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:46:49.911465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:49.911427    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:49.912120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:49.912024    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:49.927101 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:49.927066    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 22:46:49.927379 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:49.927349    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 22:46:50.115537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:50.115478    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg" event={"ID":"83416f6e-dd38-4edc-80ca-6b6fa35e64ed","Type":"ContainerStarted","Data":"0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f"}
Nov 01 22:46:50.117623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:50.117600    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:50.284677 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:50.284610    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg" podStartSLOduration=5.689862587 podStartE2EDuration="7.284591202s" totalImagesPullingTime="1.594728615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:50.264321898 +0000 UTC m=+1391.853520960" watchObservedRunningTime="2026-11-01 22:46:50.284591202 +0000 UTC m=+1391.873790265"
Nov 01 22:46:50.898149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:50.898113    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:50.898360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:50.898169    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:50.914482 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:50.914449    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 22:46:50.914839 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:50.914505    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 22:46:51.897504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:51.897477    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:51.897737 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:51.897714    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:51.915336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:51.915302    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 22:46:51.915691 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:51.915349    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 22:46:52.897821 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:52.897782    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:52.898104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:52.898080    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:52.915401 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:52.915358    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 22:46:52.915774 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:52.915744    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 22:46:53.898095 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:53.898034    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:53.898273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:53.898123    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:53.914552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:53.914521    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 22:46:53.914685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:53.914578    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 22:46:54.644320 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.644289    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"]
Nov 01 22:46:54.649281 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.649261    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.700747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.700717    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b50b3403-serving-cert\""
Nov 01 22:46:54.750855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.750816    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"]
Nov 01 22:46:54.751881 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.751849    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.752017 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.751940    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dcvx\" (UniqueName: \"kubernetes.io/projected/1e3550b8-f302-4b45-ae75-e0d091e1c50f-kube-api-access-7dcvx\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.752359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.752340    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-metrics-certs\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.752465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.752385    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3550b8-f302-4b45-ae75-e0d091e1c50f-openshift-service-ca\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.854589 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.854552    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-metrics-certs\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.854761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.854618    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3550b8-f302-4b45-ae75-e0d091e1c50f-openshift-service-ca\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.854761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.854665    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.854761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.854743    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dcvx\" (UniqueName: \"kubernetes.io/projected/1e3550b8-f302-4b45-ae75-e0d091e1c50f-kube-api-access-7dcvx\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.854933 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:46:54.854847    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-serving-cert: secret "maas-api-e2e-ait-b50b3403-serving-cert" not found
Nov 01 22:46:54.854933 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:46:54.854919    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls podName:1e3550b8-f302-4b45-ae75-e0d091e1c50f nodeName:}" failed. No retries permitted until 2026-11-01 22:46:55.354896322 +0000 UTC m=+1396.944095381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls") pod "maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c" (UID: "1e3550b8-f302-4b45-ae75-e0d091e1c50f") : secret "maas-api-e2e-ait-b50b3403-serving-cert" not found
Nov 01 22:46:54.855488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.855464    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3550b8-f302-4b45-ae75-e0d091e1c50f-openshift-service-ca\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.857245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.857226    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-metrics-certs\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:54.932721 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.932681    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:46:54.933110 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.933061    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:46:54.933651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.933628    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 22:46:54.940324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.940290    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 22:46:54.960205 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:54.960099    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dcvx\" (UniqueName: \"kubernetes.io/projected/1e3550b8-f302-4b45-ae75-e0d091e1c50f-kube-api-access-7dcvx\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:55.408320 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:55.393395    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:55.418089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:55.418053    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls\") pod \"maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:55.567234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:55.567195    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:56.058620 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:56.058589    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"]
Nov 01 22:46:56.425476 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:56.425445    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:56.426283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:56.426260    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:56.426857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:56.426836    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c" event={"ID":"1e3550b8-f302-4b45-ae75-e0d091e1c50f","Type":"ContainerStarted","Data":"99af703c962ec9b930946a29974bc81400bf3fdab3050498337c4eaf94190943"}
Nov 01 22:46:56.427112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:56.427094    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 22:46:56.549392 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:56.549365    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:56.549784 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:56.549760    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:56.550543 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:56.550524    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 22:46:57.189248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:57.189223    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:46:57.443031 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:57.442986    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c" event={"ID":"1e3550b8-f302-4b45-ae75-e0d091e1c50f","Type":"ContainerStarted","Data":"78db4e601d4c327275c9f952b963e5e3241e56f691b2895f7963d634e6e33c43"}
Nov 01 22:46:57.443227 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:57.443146    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:46:57.554381 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:46:57.554324    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c" podStartSLOduration=2.591102148 podStartE2EDuration="3.55430922s" totalImagesPullingTime="963.207072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:57.552003837 +0000 UTC m=+1399.141202898" watchObservedRunningTime="2026-11-01 22:46:57.55430922 +0000 UTC m=+1399.143508310"
Nov 01 22:47:03.456139 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:03.456109    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:47:18.816971 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:47:18.816937    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:47:40.889348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:40.889322    4837 generic.go:356] "Generic (PLEG): container finished" podID="0c518274-3546-42b4-8da0-04fcb26802bf" containerID="fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751" exitCode=0
Nov 01 22:47:40.889706 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:40.889372    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" event={"ID":"0c518274-3546-42b4-8da0-04fcb26802bf","Type":"ContainerDied","Data":"fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751"}
Nov 01 22:47:40.889755 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:40.889741    4837 scope.go:122] "RemoveContainer" containerID="fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751"
Nov 01 22:47:40.891420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:40.891388    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:40.891500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:40.891463    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:40.891539 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:40.891499    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:41.494479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.494441    4837 patch_prober.go:29] interesting pod/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:47:41.494645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.494505    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:47:41.901923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.901839    4837 generic.go:356] "Generic (PLEG): container finished" podID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerID="077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f" exitCode=0
Nov 01 22:47:41.901923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.901905    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" event={"ID":"e23ce78c-1db4-46e2-b04c-31c56f139dca","Type":"ContainerDied","Data":"077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f"}
Nov 01 22:47:41.902498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.902395    4837 scope.go:122] "RemoveContainer" containerID="077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f"
Nov 01 22:47:41.904177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.904144    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:41.904303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.904227    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:41.904303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.904265    4837 generic.go:356] "Generic (PLEG): container finished" podID="600d0c78-8fc8-4f44-9b93-96951093c210" containerID="a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab" exitCode=0
Nov 01 22:47:41.904303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.904285    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" event={"ID":"600d0c78-8fc8-4f44-9b93-96951093c210","Type":"ContainerDied","Data":"a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab"}
Nov 01 22:47:41.904457 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.904270    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:41.904637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.904619    4837 scope.go:122] "RemoveContainer" containerID="a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab"
Nov 01 22:47:41.906172 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.906142    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:41.906267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.906216    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:41.906333 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.906269    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:47:41.907375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:41.907356    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" event={"ID":"0c518274-3546-42b4-8da0-04fcb26802bf","Type":"ContainerStarted","Data":"a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08"}
Nov 01 22:47:42.605268 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:42.605240    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:47:42.606269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:42.606241    4837 patch_prober.go:29] interesting pod/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:47:42.606381 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:42.606291    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 01 22:47:42.935070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:42.934469    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" event={"ID":"e23ce78c-1db4-46e2-b04c-31c56f139dca","Type":"ContainerStarted","Data":"8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e"}
Nov 01 22:47:42.960068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:42.940366    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" event={"ID":"600d0c78-8fc8-4f44-9b93-96951093c210","Type":"ContainerStarted","Data":"bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779"}
Nov 01 22:47:43.608068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:43.607956    4837 patch_prober.go:29] interesting pod/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:47:43.608068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:43.608017    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 01 22:47:43.752066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:43.751448    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:47:43.752066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:43.751577    4837 patch_prober.go:29] interesting pod/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:47:43.752066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:43.751640    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:47:43.871078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:43.870971    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:47:43.877451 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:43.877139    4837 patch_prober.go:29] interesting pod/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 22:47:43.877451 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:43.877195    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 22:47:44.618072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:44.608467    4837 patch_prober.go:29] interesting pod/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:47:44.618072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:44.608551    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 01 22:47:44.686119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:44.682173    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:47:44.742066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:44.741570    4837 patch_prober.go:29] interesting pod/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:47:44.742066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:44.741630    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:47:44.884127 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:44.882735    4837 patch_prober.go:29] interesting pod/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 22:47:44.884127 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:44.882800    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 22:47:45.755069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:45.735260    4837 patch_prober.go:29] interesting pod/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:47:45.755069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:45.735340    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:47:45.780068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:45.778158    4837 patch_prober.go:29] interesting pod/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:47:45.780068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:45.778233    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 01 22:47:45.876070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:45.875531    4837 patch_prober.go:29] interesting pod/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 22:47:45.876070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:45.875598    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 22:47:46.612069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:46.607004    4837 patch_prober.go:29] interesting pod/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:47:46.612069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:46.607083    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 01 22:47:46.738070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:46.735002    4837 patch_prober.go:29] interesting pod/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:47:46.738070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:46.735081    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:47:46.875553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:46.875467    4837 patch_prober.go:29] interesting pod/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 22:47:46.889653 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:46.889610    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 22:47:47.604961 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:47.604915    4837 patch_prober.go:29] interesting pod/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:47:47.605177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:47.604979    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 01 22:47:47.738068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:47.734359    4837 patch_prober.go:29] interesting pod/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:47:47.738068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:47.734426    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:47:47.882907 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:47.882819    4837 patch_prober.go:29] interesting pod/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 22:47:47.883495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:47.883462    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 22:47:48.609586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:48.609557    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:47:48.610678 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:48.610644    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:47:48.721056 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:48.721013    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:47:48.747675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:48.747642    4837 patch_prober.go:29] interesting pod/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:47:48.747930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:48.747899    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:47:48.814610 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:47:48.814573    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:47:48.869633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:48.869516    4837 patch_prober.go:29] interesting pod/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 01 22:47:48.869633 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:48.869571    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 01 22:47:49.738726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:49.738695    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:47:49.739136 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:49.739098    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:47:49.739800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:49.739776    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:47:49.873998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:49.873973    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:47:49.874980 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:49.874961    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:47:50.962064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:50.956151    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"]
Nov 01 22:47:55.957252 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:55.957215    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9" podUID="9f545de2-43d8-43cc-95ee-2ced75d4e05d" containerName="payload-pre-processing" containerID="cri-o://a1b4211aff94ba8745888b109cf3e337aa300e21a03a45e6125d19d334958de8" gracePeriod=25
Nov 01 22:47:56.201398 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.201362    4837 generic.go:356] "Generic (PLEG): container finished" podID="9f545de2-43d8-43cc-95ee-2ced75d4e05d" containerID="a1b4211aff94ba8745888b109cf3e337aa300e21a03a45e6125d19d334958de8" exitCode=0
Nov 01 22:47:56.201573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.201532    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9" event={"ID":"9f545de2-43d8-43cc-95ee-2ced75d4e05d","Type":"ContainerDied","Data":"a1b4211aff94ba8745888b109cf3e337aa300e21a03a45e6125d19d334958de8"}
Nov 01 22:47:56.251092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.251068    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:47:56.427027 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.426412    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f545de2-43d8-43cc-95ee-2ced75d4e05d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f545de2-43d8-43cc-95ee-2ced75d4e05d-plugins-config-volume\") pod \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\" (UID: \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\") "
Nov 01 22:47:56.427027 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.426632    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f545de2-43d8-43cc-95ee-2ced75d4e05d-kube-api-access-76tfh\" (UniqueName: \"kubernetes.io/projected/9f545de2-43d8-43cc-95ee-2ced75d4e05d-kube-api-access-76tfh\") pod \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\" (UID: \"9f545de2-43d8-43cc-95ee-2ced75d4e05d\") "
Nov 01 22:47:56.430737 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.430700    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f545de2-43d8-43cc-95ee-2ced75d4e05d-plugins-config-volume" pod "9f545de2-43d8-43cc-95ee-2ced75d4e05d" (UID: "9f545de2-43d8-43cc-95ee-2ced75d4e05d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:47:56.431593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.431556    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f545de2-43d8-43cc-95ee-2ced75d4e05d-kube-api-access-76tfh" pod "9f545de2-43d8-43cc-95ee-2ced75d4e05d" (UID: "9f545de2-43d8-43cc-95ee-2ced75d4e05d"). InnerVolumeSpecName "kube-api-access-76tfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:47:56.529660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.529577    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f545de2-43d8-43cc-95ee-2ced75d4e05d-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:56.529660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:56.529611    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76tfh\" (UniqueName: \"kubernetes.io/projected/9f545de2-43d8-43cc-95ee-2ced75d4e05d-kube-api-access-76tfh\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:57.210070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:57.210012    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"
Nov 01 22:47:57.210513 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:57.210077    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9" event={"ID":"9f545de2-43d8-43cc-95ee-2ced75d4e05d","Type":"ContainerDied","Data":"546fa9f9f33fd4d7c9edee849de46db89ce4d7f6ecad78b47c198a0e36fed3b8"}
Nov 01 22:47:57.210513 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:57.210141    4837 scope.go:122] "RemoveContainer" containerID="a1b4211aff94ba8745888b109cf3e337aa300e21a03a45e6125d19d334958de8"
Nov 01 22:47:57.369494 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:57.369462    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"]
Nov 01 22:47:57.415513 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:57.415484    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d5014777-58486d7c47-hhcw9"]
Nov 01 22:47:58.529354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:47:58.529323    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f545de2-43d8-43cc-95ee-2ced75d4e05d" path="/var/lib/kubelet/pods/9f545de2-43d8-43cc-95ee-2ced75d4e05d/volumes"
Nov 01 22:48:10.276762 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:10.276729    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:48:11.888490 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:11.888453    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"]
Nov 01 22:48:16.889941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:16.889891    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf" podUID="1d631fdf-76e9-46b4-afd2-81104f146f07" containerName="payload-pre-processing" containerID="cri-o://51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd" gracePeriod=25
Nov 01 22:48:17.180098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.180062    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:48:17.221314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.221290    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d631fdf-76e9-46b4-afd2-81104f146f07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d631fdf-76e9-46b4-afd2-81104f146f07-plugins-config-volume\") pod \"1d631fdf-76e9-46b4-afd2-81104f146f07\" (UID: \"1d631fdf-76e9-46b4-afd2-81104f146f07\") "
Nov 01 22:48:17.221487 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.221474    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d631fdf-76e9-46b4-afd2-81104f146f07-kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/1d631fdf-76e9-46b4-afd2-81104f146f07-kube-api-access-2nbdd\") pod \"1d631fdf-76e9-46b4-afd2-81104f146f07\" (UID: \"1d631fdf-76e9-46b4-afd2-81104f146f07\") "
Nov 01 22:48:17.221669 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.221645    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d631fdf-76e9-46b4-afd2-81104f146f07-plugins-config-volume" pod "1d631fdf-76e9-46b4-afd2-81104f146f07" (UID: "1d631fdf-76e9-46b4-afd2-81104f146f07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:17.222029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.222011    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d631fdf-76e9-46b4-afd2-81104f146f07-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:17.223484 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.223465    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d631fdf-76e9-46b4-afd2-81104f146f07-kube-api-access-2nbdd" pod "1d631fdf-76e9-46b4-afd2-81104f146f07" (UID: "1d631fdf-76e9-46b4-afd2-81104f146f07"). InnerVolumeSpecName "kube-api-access-2nbdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:17.323676 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.323640    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nbdd\" (UniqueName: \"kubernetes.io/projected/1d631fdf-76e9-46b4-afd2-81104f146f07-kube-api-access-2nbdd\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:17.603607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.603578    4837 generic.go:356] "Generic (PLEG): container finished" podID="1d631fdf-76e9-46b4-afd2-81104f146f07" containerID="51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd" exitCode=0
Nov 01 22:48:17.603785 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.603622    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"
Nov 01 22:48:17.603785 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.603664    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf" event={"ID":"1d631fdf-76e9-46b4-afd2-81104f146f07","Type":"ContainerDied","Data":"51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd"}
Nov 01 22:48:17.603785 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.603717    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf" event={"ID":"1d631fdf-76e9-46b4-afd2-81104f146f07","Type":"ContainerDied","Data":"058ec4d10b4b522969184cc3302ae9810a16801d04ceea357f4246828fc23225"}
Nov 01 22:48:17.603785 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.603746    4837 scope.go:122] "RemoveContainer" containerID="51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd"
Nov 01 22:48:17.615168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.615148    4837 scope.go:122] "RemoveContainer" containerID="51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd"
Nov 01 22:48:17.615427 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:17.615407    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd\": container with ID starting with 51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd not found: ID does not exist" containerID="51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd"
Nov 01 22:48:17.615484 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.615435    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd"} err="failed to get container status \"51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd\": rpc error: code = NotFound desc = could not find container \"51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd\": container with ID starting with 51bea0d9031b64df6c6c38b118feaa98bf20b1951197db08a1fa0175c08e17dd not found: ID does not exist"
Nov 01 22:48:17.768181 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.768156    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"]
Nov 01 22:48:17.814261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:17.814202    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b50b3403-7c4679fd6c-tmpxf"]
Nov 01 22:48:18.528263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:18.528231    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d631fdf-76e9-46b4-afd2-81104f146f07" path="/var/lib/kubelet/pods/1d631fdf-76e9-46b4-afd2-81104f146f07/volumes"
Nov 01 22:48:18.813270 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:18.813192    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:48:25.612663 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:25.612631    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"]
Nov 01 22:48:25.613205 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:25.612862    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7" podUID="0e34436a-4e94-4692-85ad-7cd548e952ad" containerName="maas-api" containerID="cri-o://2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481" gracePeriod=30
Nov 01 22:48:26.105139 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.099262    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:48:26.321284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.312132    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-metrics-certs\") pod \"0e34436a-4e94-4692-85ad-7cd548e952ad\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") "
Nov 01 22:48:26.321284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.315244    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls\") pod \"0e34436a-4e94-4692-85ad-7cd548e952ad\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") "
Nov 01 22:48:26.323397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.323364    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e34436a-4e94-4692-85ad-7cd548e952ad-kube-api-access-kbtsh\" (UniqueName: \"kubernetes.io/projected/0e34436a-4e94-4692-85ad-7cd548e952ad-kube-api-access-kbtsh\") pod \"0e34436a-4e94-4692-85ad-7cd548e952ad\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") "
Nov 01 22:48:26.323914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.323890    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e34436a-4e94-4692-85ad-7cd548e952ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34436a-4e94-4692-85ad-7cd548e952ad-openshift-service-ca\") pod \"0e34436a-4e94-4692-85ad-7cd548e952ad\" (UID: \"0e34436a-4e94-4692-85ad-7cd548e952ad\") "
Nov 01 22:48:26.324325 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.324278    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e34436a-4e94-4692-85ad-7cd548e952ad-openshift-service-ca" pod "0e34436a-4e94-4692-85ad-7cd548e952ad" (UID: "0e34436a-4e94-4692-85ad-7cd548e952ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:26.342074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.341647    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34436a-4e94-4692-85ad-7cd548e952ad-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:26.367977 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.367767    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e34436a-4e94-4692-85ad-7cd548e952ad-kube-api-access-kbtsh" pod "0e34436a-4e94-4692-85ad-7cd548e952ad" (UID: "0e34436a-4e94-4692-85ad-7cd548e952ad"). InnerVolumeSpecName "kube-api-access-kbtsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:26.386135 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.384387    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls" pod "0e34436a-4e94-4692-85ad-7cd548e952ad" (UID: "0e34436a-4e94-4692-85ad-7cd548e952ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:26.386135 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.384533    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-metrics-certs" pod "0e34436a-4e94-4692-85ad-7cd548e952ad" (UID: "0e34436a-4e94-4692-85ad-7cd548e952ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:26.443786 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.443753    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:26.443786 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.443783    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34436a-4e94-4692-85ad-7cd548e952ad-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:26.443990 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.443796    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbtsh\" (UniqueName: \"kubernetes.io/projected/0e34436a-4e94-4692-85ad-7cd548e952ad-kube-api-access-kbtsh\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:26.751067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.747570    4837 generic.go:356] "Generic (PLEG): container finished" podID="0e34436a-4e94-4692-85ad-7cd548e952ad" containerID="2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481" exitCode=0
Nov 01 22:48:26.751067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.748002    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"
Nov 01 22:48:26.759067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.752198    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7" event={"ID":"0e34436a-4e94-4692-85ad-7cd548e952ad","Type":"ContainerDied","Data":"2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481"}
Nov 01 22:48:26.759067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.752344    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7" event={"ID":"0e34436a-4e94-4692-85ad-7cd548e952ad","Type":"ContainerDied","Data":"00efa7a6f5f266482da7523252aa73e95b8878df68d418c80993c3e68fcaf79a"}
Nov 01 22:48:26.759067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.752466    4837 scope.go:122] "RemoveContainer" containerID="2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481"
Nov 01 22:48:26.892911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.880471    4837 scope.go:122] "RemoveContainer" containerID="2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481"
Nov 01 22:48:26.892911 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:26.892328    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481\": container with ID starting with 2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481 not found: ID does not exist" containerID="2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481"
Nov 01 22:48:26.892911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.892369    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481"} err="failed to get container status \"2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481\": rpc error: code = NotFound desc = could not find container \"2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481\": container with ID starting with 2aabce87f240f79cb0469c3f0beca54930c105ef2bb01111b9bbc986760d1481 not found: ID does not exist"
Nov 01 22:48:26.938465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.938432    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"]
Nov 01 22:48:26.984697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:26.984661    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5014777-55b5cfff58-8bvr7"]
Nov 01 22:48:27.989802 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:27.989767    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"]
Nov 01 22:48:27.990277 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:27.990101    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy" containerID="cri-o://a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08" gracePeriod=30
Nov 01 22:48:28.528399 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:28.528368    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e34436a-4e94-4692-85ad-7cd548e952ad" path="/var/lib/kubelet/pods/0e34436a-4e94-4692-85ad-7cd548e952ad/volumes"
Nov 01 22:48:31.003858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.003825    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"]
Nov 01 22:48:31.004561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004547    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1d631fdf-76e9-46b4-afd2-81104f146f07" containerName="payload-pre-processing"
Nov 01 22:48:31.004607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004562    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d631fdf-76e9-46b4-afd2-81104f146f07" containerName="payload-pre-processing"
Nov 01 22:48:31.004607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004577    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9f545de2-43d8-43cc-95ee-2ced75d4e05d" containerName="payload-pre-processing"
Nov 01 22:48:31.004607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004582    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f545de2-43d8-43cc-95ee-2ced75d4e05d" containerName="payload-pre-processing"
Nov 01 22:48:31.004607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004603    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0e34436a-4e94-4692-85ad-7cd548e952ad" containerName="maas-api"
Nov 01 22:48:31.004607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004607    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e34436a-4e94-4692-85ad-7cd548e952ad" containerName="maas-api"
Nov 01 22:48:31.004757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004711    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d631fdf-76e9-46b4-afd2-81104f146f07" containerName="payload-pre-processing"
Nov 01 22:48:31.004757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004721    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f545de2-43d8-43cc-95ee-2ced75d4e05d" containerName="payload-pre-processing"
Nov 01 22:48:31.004757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.004730    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e34436a-4e94-4692-85ad-7cd548e952ad" containerName="maas-api"
Nov 01 22:48:31.009622 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.009598    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.109180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.109146    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"]
Nov 01 22:48:31.122941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.122905    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-credential-socket\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.123158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.123011    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-data\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.123324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.123300    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-socket\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.123371 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.123340    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx492\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-kube-api-access-xx492\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.123431 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.123414    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-envoy\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.123518 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.123503    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-istio-token\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.123589 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.123572    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7646a63-328d-4629-bd52-0f0f50486d82-istio-podinfo\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.123651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.123617    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7646a63-328d-4629-bd52-0f0f50486d82-istiod-ca-cert\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.123651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.123643    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-certs\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.196067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.158261    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d04c71aa-gw-openshift-default-dockercfg-l8dqq\""
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.252382    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-socket\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.263504    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx492\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-kube-api-access-xx492\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.263578    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-envoy\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.263651    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-istio-token\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.263704    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7646a63-328d-4629-bd52-0f0f50486d82-istio-podinfo\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.263739    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7646a63-328d-4629-bd52-0f0f50486d82-istiod-ca-cert\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.263766    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-certs\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.263807    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-credential-socket\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.263832    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-data\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.264204    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-data\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.252816    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-socket\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.265436    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7646a63-328d-4629-bd52-0f0f50486d82-istiod-ca-cert\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.266363    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-certs\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.276766 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.266558    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-credential-socket\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.306064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.296379    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-envoy\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.374234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.374197    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7646a63-328d-4629-bd52-0f0f50486d82-istio-podinfo\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.382998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.382965    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-istio-token\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.429366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.428612    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx492\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-kube-api-access-xx492\") pod \"e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") " pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:31.624339 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:31.624258    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:32.047401 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:48:32.047374    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7646a63_328d_4629_bd52_0f0f50486d82.slice/crio-16528e40478aee03593079bb90086a581875418cc2541012e678e62fc2bf7d22 WatchSource:0}: Error finding container 16528e40478aee03593079bb90086a581875418cc2541012e678e62fc2bf7d22: Status 404 returned error can't find the container with id 16528e40478aee03593079bb90086a581875418cc2541012e678e62fc2bf7d22
Nov 01 22:48:32.048837 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.048806    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"]
Nov 01 22:48:32.049950 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.049918    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:32.050018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.049985    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:32.050018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.050014    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:32.226166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.226131    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"]
Nov 01 22:48:32.226355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.226339    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg" podUID="83416f6e-dd38-4edc-80ca-6b6fa35e64ed" containerName="maas-api" containerID="cri-o://0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f" gracePeriod=30
Nov 01 22:48:32.800065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.788386    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:48:32.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.882239    4837 generic.go:356] "Generic (PLEG): container finished" podID="83416f6e-dd38-4edc-80ca-6b6fa35e64ed" containerID="0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f" exitCode=0
Nov 01 22:48:32.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.882339    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg" event={"ID":"83416f6e-dd38-4edc-80ca-6b6fa35e64ed","Type":"ContainerDied","Data":"0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f"}
Nov 01 22:48:32.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.882389    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg" event={"ID":"83416f6e-dd38-4edc-80ca-6b6fa35e64ed","Type":"ContainerDied","Data":"9d0566d55538c1ede0cd99a35821dff359ebe69b1c895a164a30b32fdaca3b49"}
Nov 01 22:48:32.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.882421    4837 scope.go:122] "RemoveContainer" containerID="0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f"
Nov 01 22:48:32.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.882576    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"
Nov 01 22:48:32.897639 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.897565    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" event={"ID":"e7646a63-328d-4629-bd52-0f0f50486d82","Type":"ContainerStarted","Data":"d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8"}
Nov 01 22:48:32.897916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.897868    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" event={"ID":"e7646a63-328d-4629-bd52-0f0f50486d82","Type":"ContainerStarted","Data":"16528e40478aee03593079bb90086a581875418cc2541012e678e62fc2bf7d22"}
Nov 01 22:48:32.927975 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.927940    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-metrics-certs\") pod \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") "
Nov 01 22:48:32.952349 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.952267    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-maas-api-tls\") pod \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") "
Nov 01 22:48:32.952843 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.952819    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-openshift-service-ca\") pod \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") "
Nov 01 22:48:32.953072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.953034    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-kube-api-access-v6mxq\" (UniqueName: \"kubernetes.io/projected/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-kube-api-access-v6mxq\") pod \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\" (UID: \"83416f6e-dd38-4edc-80ca-6b6fa35e64ed\") "
Nov 01 22:48:32.955261 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.955216    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-openshift-service-ca" pod "83416f6e-dd38-4edc-80ca-6b6fa35e64ed" (UID: "83416f6e-dd38-4edc-80ca-6b6fa35e64ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:32.966969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.966940    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-maas-api-tls" pod "83416f6e-dd38-4edc-80ca-6b6fa35e64ed" (UID: "83416f6e-dd38-4edc-80ca-6b6fa35e64ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:32.973353 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.973316    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-kube-api-access-v6mxq" pod "83416f6e-dd38-4edc-80ca-6b6fa35e64ed" (UID: "83416f6e-dd38-4edc-80ca-6b6fa35e64ed"). InnerVolumeSpecName "kube-api-access-v6mxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:32.986459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:32.986429    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-metrics-certs" pod "83416f6e-dd38-4edc-80ca-6b6fa35e64ed" (UID: "83416f6e-dd38-4edc-80ca-6b6fa35e64ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:33.000130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.000102    4837 scope.go:122] "RemoveContainer" containerID="0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f"
Nov 01 22:48:33.000496 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:33.000458    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f\": container with ID starting with 0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f not found: ID does not exist" containerID="0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f"
Nov 01 22:48:33.000561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.000512    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f"} err="failed to get container status \"0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f\": rpc error: code = NotFound desc = could not find container \"0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f\": container with ID starting with 0f105d9f2b3999a72680c7d13ac38a55c8c89e9412a4df0ade9321822df9529f not found: ID does not exist"
Nov 01 22:48:33.060538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.060469    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" podStartSLOduration=3.060449198 podStartE2EDuration="3.060449198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:33.058439484 +0000 UTC m=+1494.647638548" watchObservedRunningTime="2026-11-01 22:48:33.060449198 +0000 UTC m=+1494.649648261"
Nov 01 22:48:33.063189 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.063162    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.063356 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.063339    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mxq\" (UniqueName: \"kubernetes.io/projected/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-kube-api-access-v6mxq\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.063457 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.063437    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.063457 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.063457    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83416f6e-dd38-4edc-80ca-6b6fa35e64ed-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.624995 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.624968    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:33.632777 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.632753    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:33.676972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.676945    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:48:33.686279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.686241    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-socket\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.686437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.686306    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c518274-3546-42b4-8da0-04fcb26802bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c518274-3546-42b4-8da0-04fcb26802bf-istiod-ca-cert\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.686437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.686362    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c518274-3546-42b4-8da0-04fcb26802bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c518274-3546-42b4-8da0-04fcb26802bf-istio-podinfo\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.686544 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.686432    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-socket" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:33.686797 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.686760    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c518274-3546-42b4-8da0-04fcb26802bf-istiod-ca-cert" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:33.687054 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.687017    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.687156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.687080    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c518274-3546-42b4-8da0-04fcb26802bf-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.689254 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.689231    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c518274-3546-42b4-8da0-04fcb26802bf-istio-podinfo" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:48:33.711812 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.711789    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"]
Nov 01 22:48:33.757681 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.757655    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-798f4550-7cd5fcd849-swhlg"]
Nov 01 22:48:33.788474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788447    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-credential-socket\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.788587 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788564    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-certs\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.788636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788591    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-kube-api-access-cblgz\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-kube-api-access-cblgz\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.788636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788625    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-envoy\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.788752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788726    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-credential-socket" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:33.788819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788751    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-certs" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:33.788819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788755    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-data\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.788931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788871    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-istio-token\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-istio-token\") pod \"0c518274-3546-42b4-8da0-04fcb26802bf\" (UID: \"0c518274-3546-42b4-8da0-04fcb26802bf\") "
Nov 01 22:48:33.789027 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.788974    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-data" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:33.789406 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.789382    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.789516 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.789411    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.789516 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.789426    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c518274-3546-42b4-8da0-04fcb26802bf-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.789516 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.789436    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.790960 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.790941    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-kube-api-access-cblgz" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "kube-api-access-cblgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:33.791343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.791324    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-envoy" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:33.791414 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.791362    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-istio-token" pod "0c518274-3546-42b4-8da0-04fcb26802bf" (UID: "0c518274-3546-42b4-8da0-04fcb26802bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:33.890874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.890800    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblgz\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-kube-api-access-cblgz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.890874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.890837    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c518274-3546-42b4-8da0-04fcb26802bf-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.890874 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.890849    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c518274-3546-42b4-8da0-04fcb26802bf-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:33.908498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.908468    4837 generic.go:356] "Generic (PLEG): container finished" podID="0c518274-3546-42b4-8da0-04fcb26802bf" containerID="a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08" exitCode=0
Nov 01 22:48:33.908648 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.908523    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"
Nov 01 22:48:33.908648 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.908550    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" event={"ID":"0c518274-3546-42b4-8da0-04fcb26802bf","Type":"ContainerDied","Data":"a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08"}
Nov 01 22:48:33.908648 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.908623    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" event={"ID":"0c518274-3546-42b4-8da0-04fcb26802bf","Type":"ContainerDied","Data":"8b36a2b0ffb76f5963bdea43830313cc747542600787dac9395fb588ec32ae09"}
Nov 01 22:48:33.908648 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.908647    4837 scope.go:122] "RemoveContainer" containerID="a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08"
Nov 01 22:48:33.908941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.908918    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:33.910252 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.909894    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:48:33.921866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.921845    4837 scope.go:122] "RemoveContainer" containerID="fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751"
Nov 01 22:48:33.935229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.935205    4837 scope.go:122] "RemoveContainer" containerID="a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08"
Nov 01 22:48:33.935521 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:33.935503    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08\": container with ID starting with a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08 not found: ID does not exist" containerID="a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08"
Nov 01 22:48:33.935594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.935530    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08"} err="failed to get container status \"a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08\": rpc error: code = NotFound desc = could not find container \"a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08\": container with ID starting with a70d8deb7b07eb1c27d499b72a5a3096965010115d777c8b6b2539c67ddbfb08 not found: ID does not exist"
Nov 01 22:48:33.935594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.935549    4837 scope.go:122] "RemoveContainer" containerID="fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751"
Nov 01 22:48:33.935805 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:33.935789    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751\": container with ID starting with fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751 not found: ID does not exist" containerID="fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751"
Nov 01 22:48:33.935847 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:33.935810    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751"} err="failed to get container status \"fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751\": rpc error: code = NotFound desc = could not find container \"fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751\": container with ID starting with fffce590946f2c8646d0c3a0f1df96d0ad63910c249bf114a3e5e4b04b0ae751 not found: ID does not exist"
Nov 01 22:48:34.175313 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:34.175284    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"]
Nov 01 22:48:34.180802 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:34.180780    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7"]
Nov 01 22:48:34.527345 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:34.527273    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" path="/var/lib/kubelet/pods/0c518274-3546-42b4-8da0-04fcb26802bf/volumes"
Nov 01 22:48:34.527789 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:34.527773    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83416f6e-dd38-4edc-80ca-6b6fa35e64ed" path="/var/lib/kubelet/pods/83416f6e-dd38-4edc-80ca-6b6fa35e64ed/volumes"
Nov 01 22:48:34.612315 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:34.612287    4837 patch_prober.go:29] interesting pod/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:48:34.612474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:34.612331    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d5014777-openshift-default-644c7559cf-9grq7" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:35.793212 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:35.793183    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"]
Nov 01 22:48:35.793692 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:35.793401    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy" containerID="cri-o://bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779" gracePeriod=30
Nov 01 22:48:37.098063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.091677    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"]
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.101808    4837 cpu_manager.go:414] "containerMap: removing container" podUID="83416f6e-dd38-4edc-80ca-6b6fa35e64ed" containerName="maas-api"
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.101835    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83416f6e-dd38-4edc-80ca-6b6fa35e64ed" containerName="maas-api"
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.101880    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy"
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.101889    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy"
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.101902    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy"
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.101909    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy"
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.102171    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="83416f6e-dd38-4edc-80ca-6b6fa35e64ed" containerName="maas-api"
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.102193    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy"
Nov 01 22:48:37.105538 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.102204    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c518274-3546-42b4-8da0-04fcb26802bf" containerName="istio-proxy"
Nov 01 22:48:37.215226 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.119395    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.304018    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-68a7f708-openshift-default-dockercfg-5kpj5\""
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.305794    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istiod-ca-cert\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.305924    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-socket\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.305969    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq8dl\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-kube-api-access-bq8dl\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.306069    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-envoy\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.306094    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-token\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.306170    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-podinfo\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.306232    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-certs\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.306293    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-data\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.377119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.306360    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-credential-socket\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.421572    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-podinfo\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.421675    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-certs\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.421745    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-data\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.421819    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-credential-socket\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.421994    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istiod-ca-cert\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.422032    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-socket\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.431586    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8dl\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-kube-api-access-bq8dl\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.431759    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-envoy\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.433065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.431799    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-token\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.460229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.442116    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"]
Nov 01 22:48:37.460229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.443409    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-data\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.460229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.431012    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-certs\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.460229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.454197    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-socket\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.460229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.430733    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-credential-socket\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.464063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.460864    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istiod-ca-cert\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.516065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.514602    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-envoy\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.587868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.586745    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-podinfo\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.591455 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.591416    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-token\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.600091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.599080    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8dl\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-kube-api-access-bq8dl\") pod \"e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") " pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:37.869206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:37.869173    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:38.297626 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:48:38.297594    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7119d22b_44f1_4c43_8a7b_c1dc954edaab.slice/crio-525bbe5cd1621be2ceeed473b03cbfc7ed19b3a7646c28ba0b07a2cb0b7ab64e WatchSource:0}: Error finding container 525bbe5cd1621be2ceeed473b03cbfc7ed19b3a7646c28ba0b07a2cb0b7ab64e: Status 404 returned error can't find the container with id 525bbe5cd1621be2ceeed473b03cbfc7ed19b3a7646c28ba0b07a2cb0b7ab64e
Nov 01 22:48:38.298491 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:38.298467    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"]
Nov 01 22:48:38.300487 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:38.300458    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:38.300556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:38.300519    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:38.300595 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:38.300554    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:38.715195 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:38.715165    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:48:38.719827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:38.719803    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:48:38.885903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:38.885878    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"]
Nov 01 22:48:38.886150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:38.886127    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c" podUID="1e3550b8-f302-4b45-ae75-e0d091e1c50f" containerName="maas-api" containerID="cri-o://78db4e601d4c327275c9f952b963e5e3241e56f691b2895f7963d634e6e33c43" gracePeriod=30
Nov 01 22:48:39.114647 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.114613    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" event={"ID":"7119d22b-44f1-4c43-8a7b-c1dc954edaab","Type":"ContainerStarted","Data":"8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62"}
Nov 01 22:48:39.114903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.114678    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" event={"ID":"7119d22b-44f1-4c43-8a7b-c1dc954edaab","Type":"ContainerStarted","Data":"525bbe5cd1621be2ceeed473b03cbfc7ed19b3a7646c28ba0b07a2cb0b7ab64e"}
Nov 01 22:48:39.116655 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.116623    4837 generic.go:356] "Generic (PLEG): container finished" podID="1e3550b8-f302-4b45-ae75-e0d091e1c50f" containerID="78db4e601d4c327275c9f952b963e5e3241e56f691b2895f7963d634e6e33c43" exitCode=0
Nov 01 22:48:39.116755 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.116692    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c" event={"ID":"1e3550b8-f302-4b45-ae75-e0d091e1c50f","Type":"ContainerDied","Data":"78db4e601d4c327275c9f952b963e5e3241e56f691b2895f7963d634e6e33c43"}
Nov 01 22:48:39.245835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.244219    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" podStartSLOduration=3.244196831 podStartE2EDuration="3.244196831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:39.237473444 +0000 UTC m=+1500.826672505" watchObservedRunningTime="2026-11-01 22:48:39.244196831 +0000 UTC m=+1500.833395893"
Nov 01 22:48:39.319083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.318637    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:48:39.476756 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.476722    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3550b8-f302-4b45-ae75-e0d091e1c50f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3550b8-f302-4b45-ae75-e0d091e1c50f-openshift-service-ca\") pod \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") "
Nov 01 22:48:39.477161 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.477038    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3550b8-f302-4b45-ae75-e0d091e1c50f-kube-api-access-7dcvx\" (UniqueName: \"kubernetes.io/projected/1e3550b8-f302-4b45-ae75-e0d091e1c50f-kube-api-access-7dcvx\") pod \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") "
Nov 01 22:48:39.477356 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.477339    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-metrics-certs\") pod \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") "
Nov 01 22:48:39.477525 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.477510    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls\") pod \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\" (UID: \"1e3550b8-f302-4b45-ae75-e0d091e1c50f\") "
Nov 01 22:48:39.488233 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.488195    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3550b8-f302-4b45-ae75-e0d091e1c50f-kube-api-access-7dcvx" pod "1e3550b8-f302-4b45-ae75-e0d091e1c50f" (UID: "1e3550b8-f302-4b45-ae75-e0d091e1c50f"). InnerVolumeSpecName "kube-api-access-7dcvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:39.494111 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.488537    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3550b8-f302-4b45-ae75-e0d091e1c50f-openshift-service-ca" pod "1e3550b8-f302-4b45-ae75-e0d091e1c50f" (UID: "1e3550b8-f302-4b45-ae75-e0d091e1c50f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:39.535652 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.535610    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-metrics-certs" pod "1e3550b8-f302-4b45-ae75-e0d091e1c50f" (UID: "1e3550b8-f302-4b45-ae75-e0d091e1c50f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:39.570886 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.570796    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls" pod "1e3550b8-f302-4b45-ae75-e0d091e1c50f" (UID: "1e3550b8-f302-4b45-ae75-e0d091e1c50f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:39.580229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.580204    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3550b8-f302-4b45-ae75-e0d091e1c50f-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:39.580361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.580230    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dcvx\" (UniqueName: \"kubernetes.io/projected/1e3550b8-f302-4b45-ae75-e0d091e1c50f-kube-api-access-7dcvx\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:39.580361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.580256    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:39.580361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.580264    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3550b8-f302-4b45-ae75-e0d091e1c50f-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:39.647032 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.647003    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"]
Nov 01 22:48:39.647714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.647699    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1e3550b8-f302-4b45-ae75-e0d091e1c50f" containerName="maas-api"
Nov 01 22:48:39.647753 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.647717    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3550b8-f302-4b45-ae75-e0d091e1c50f" containerName="maas-api"
Nov 01 22:48:39.647837 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.647829    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3550b8-f302-4b45-ae75-e0d091e1c50f" containerName="maas-api"
Nov 01 22:48:39.651476 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.651459    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:39.681787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.681764    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2952f9-61a3-4869-9da0-999c4caeea56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz\" (UID: \"9f2952f9-61a3-4869-9da0-999c4caeea56\") " pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:39.681952 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.681931    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbm4f\" (UniqueName: \"kubernetes.io/projected/9f2952f9-61a3-4869-9da0-999c4caeea56-kube-api-access-bbm4f\") pod \"payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz\" (UID: \"9f2952f9-61a3-4869-9da0-999c4caeea56\") " pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:39.710106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.709715    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d04c71aa-dockercfg-7c7ln\""
Nov 01 22:48:39.713388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.713226    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d04c71aa\""
Nov 01 22:48:39.717315 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.717290    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"]
Nov 01 22:48:39.828278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.821910    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2952f9-61a3-4869-9da0-999c4caeea56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz\" (UID: \"9f2952f9-61a3-4869-9da0-999c4caeea56\") " pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:39.828278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.820812    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2952f9-61a3-4869-9da0-999c4caeea56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz\" (UID: \"9f2952f9-61a3-4869-9da0-999c4caeea56\") " pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:39.828278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.822282    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbm4f\" (UniqueName: \"kubernetes.io/projected/9f2952f9-61a3-4869-9da0-999c4caeea56-kube-api-access-bbm4f\") pod \"payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz\" (UID: \"9f2952f9-61a3-4869-9da0-999c4caeea56\") " pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:39.884962 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.879613    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:39.907075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.904629    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:39.922444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:39.922405    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"]
Nov 01 22:48:40.308104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.305524    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"]
Nov 01 22:48:40.308104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.306151    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:40.322108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.320362    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbm4f\" (UniqueName: \"kubernetes.io/projected/9f2952f9-61a3-4869-9da0-999c4caeea56-kube-api-access-bbm4f\") pod \"payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz\" (UID: \"9f2952f9-61a3-4869-9da0-999c4caeea56\") " pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:40.434687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.434644    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm54r\" (UniqueName: \"kubernetes.io/projected/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-kube-api-access-rm54r\") pod \"payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt\" (UID: \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:40.435319 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.435291    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt\" (UID: \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:40.464074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.460418    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"
Nov 01 22:48:40.482029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.481365    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c" event={"ID":"1e3550b8-f302-4b45-ae75-e0d091e1c50f","Type":"ContainerDied","Data":"99af703c962ec9b930946a29974bc81400bf3fdab3050498337c4eaf94190943"}
Nov 01 22:48:40.482029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.481454    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:40.482029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.481489    4837 scope.go:122] "RemoveContainer" containerID="78db4e601d4c327275c9f952b963e5e3241e56f691b2895f7963d634e6e33c43"
Nov 01 22:48:40.498796 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.498664    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:48:40.546949 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.545156    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm54r\" (UniqueName: \"kubernetes.io/projected/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-kube-api-access-rm54r\") pod \"payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt\" (UID: \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:40.546949 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.545757    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt\" (UID: \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:40.546949 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.546927    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt\" (UID: \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:40.565915 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.565684    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:40.611929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.611900    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm54r\" (UniqueName: \"kubernetes.io/projected/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-kube-api-access-rm54r\") pod \"payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt\" (UID: \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:40.758238 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.758207    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"]
Nov 01 22:48:40.763915 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.763891    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b50b3403-7c6b9c7696-twp4c"]
Nov 01 22:48:40.870567 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.870542    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:40.966174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:40.965750    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"]
Nov 01 22:48:41.097391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.097364    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:48:41.155348 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155319    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-envoy\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.155501 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155401    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-kube-api-access-jd2rh\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-kube-api-access-jd2rh\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.155501 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155436    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600d0c78-8fc8-4f44-9b93-96951093c210-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d0c78-8fc8-4f44-9b93-96951093c210-istiod-ca-cert\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.155626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155567    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-credential-socket\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.155626 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155593    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/600d0c78-8fc8-4f44-9b93-96951093c210-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d0c78-8fc8-4f44-9b93-96951093c210-istio-podinfo\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.155739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155694    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-data\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.155828 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155809    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-istio-token\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-istio-token\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.156070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155803    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-credential-socket" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:41.156070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155895    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600d0c78-8fc8-4f44-9b93-96951093c210-istiod-ca-cert" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:41.156070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.155908    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-certs\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.156070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.156003    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-data" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:41.156372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.156083    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-socket\") pod \"600d0c78-8fc8-4f44-9b93-96951093c210\" (UID: \"600d0c78-8fc8-4f44-9b93-96951093c210\") "
Nov 01 22:48:41.156372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.156151    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-certs" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:41.156472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.156373    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-socket" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:41.157012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.156986    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.157113 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.157016    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.157113 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.157031    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.157113 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.157065    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.157113 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.157079    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d0c78-8fc8-4f44-9b93-96951093c210-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.157804 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.157774    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/600d0c78-8fc8-4f44-9b93-96951093c210-istio-podinfo" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:48:41.157897 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.157878    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-kube-api-access-jd2rh" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "kube-api-access-jd2rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:41.157940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.157915    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-envoy" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:41.158221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.158207    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-istio-token" pod "600d0c78-8fc8-4f44-9b93-96951093c210" (UID: "600d0c78-8fc8-4f44-9b93-96951093c210"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:41.258600 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.258560    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d0c78-8fc8-4f44-9b93-96951093c210-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.258820 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.258794    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.258820 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.258822    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d0c78-8fc8-4f44-9b93-96951093c210-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.259089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.258837    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2rh\" (UniqueName: \"kubernetes.io/projected/600d0c78-8fc8-4f44-9b93-96951093c210-kube-api-access-jd2rh\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:41.347580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.347495    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"]
Nov 01 22:48:41.468400 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.468374    4837 generic.go:356] "Generic (PLEG): container finished" podID="600d0c78-8fc8-4f44-9b93-96951093c210" containerID="bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779" exitCode=0
Nov 01 22:48:41.468509 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.468423    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"
Nov 01 22:48:41.468509 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.468456    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" event={"ID":"600d0c78-8fc8-4f44-9b93-96951093c210","Type":"ContainerDied","Data":"bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779"}
Nov 01 22:48:41.468625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.468520    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf" event={"ID":"600d0c78-8fc8-4f44-9b93-96951093c210","Type":"ContainerDied","Data":"5bd1d6f7266fc21aa338b15bb943b3a4323d8b06614704edf878dc1a0951735d"}
Nov 01 22:48:41.468625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.468551    4837 scope.go:122] "RemoveContainer" containerID="bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779"
Nov 01 22:48:41.471854 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.471828    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt" event={"ID":"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f","Type":"ContainerStarted","Data":"1380fff35f22af35d00ac2d9c0f24585296ea94306d69b5cd14c23ceef64c554"}
Nov 01 22:48:41.471949 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.471904    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:41.474199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.474177    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz" event={"ID":"9f2952f9-61a3-4869-9da0-999c4caeea56","Type":"ContainerStarted","Data":"37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee"}
Nov 01 22:48:41.474374 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.474352    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz" event={"ID":"9f2952f9-61a3-4869-9da0-999c4caeea56","Type":"ContainerStarted","Data":"393150eeef5db912056c0c1c5c1939154c0c1e19b6c548f0ef7e62f2da89b140"}
Nov 01 22:48:41.474478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.474388    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:41.481168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.481150    4837 scope.go:122] "RemoveContainer" containerID="a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab"
Nov 01 22:48:41.490555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.490387    4837 scope.go:122] "RemoveContainer" containerID="bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779"
Nov 01 22:48:41.490678 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:41.490659    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779\": container with ID starting with bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779 not found: ID does not exist" containerID="bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779"
Nov 01 22:48:41.490723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.490686    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779"} err="failed to get container status \"bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779\": rpc error: code = NotFound desc = could not find container \"bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779\": container with ID starting with bfaba0056211a3d854595f8b30b5c9acd5597bbef38dee5c377dca6e8a9d2779 not found: ID does not exist"
Nov 01 22:48:41.490723 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.490704    4837 scope.go:122] "RemoveContainer" containerID="a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab"
Nov 01 22:48:41.490980 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:41.490965    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab\": container with ID starting with a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab not found: ID does not exist" containerID="a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab"
Nov 01 22:48:41.491017 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.490985    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab"} err="failed to get container status \"a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab\": rpc error: code = NotFound desc = could not find container \"a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab\": container with ID starting with a0708a7a899537082c035c07543b1850bc498ce53301f05cf92f393ddb9486ab not found: ID does not exist"
Nov 01 22:48:41.629698 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.629617    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"]
Nov 01 22:48:41.636395 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.636370    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-798f4550-openshift-default-6465cf4df7-qvtbf"]
Nov 01 22:48:41.742999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.742947    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz" podStartSLOduration=2.742932298 podStartE2EDuration="2.742932298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:41.737759137 +0000 UTC m=+1503.326958199" watchObservedRunningTime="2026-11-01 22:48:41.742932298 +0000 UTC m=+1503.332131359"
Nov 01 22:48:41.856036 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.855979    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt" podStartSLOduration=2.855960979 podStartE2EDuration="2.855960979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:41.852922737 +0000 UTC m=+1503.442121797" watchObservedRunningTime="2026-11-01 22:48:41.855960979 +0000 UTC m=+1503.445160041"
Nov 01 22:48:41.872327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.872299    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"]
Nov 01 22:48:41.872517 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:41.872496    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy" containerID="cri-o://8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e" gracePeriod=30
Nov 01 22:48:42.486660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:42.486626    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt" event={"ID":"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f","Type":"ContainerStarted","Data":"8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae"}
Nov 01 22:48:42.528362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:42.528335    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3550b8-f302-4b45-ae75-e0d091e1c50f" path="/var/lib/kubelet/pods/1e3550b8-f302-4b45-ae75-e0d091e1c50f/volumes"
Nov 01 22:48:42.528804 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:42.528790    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" path="/var/lib/kubelet/pods/600d0c78-8fc8-4f44-9b93-96951093c210/volumes"
Nov 01 22:48:45.699777 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.696446    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"]
Nov 01 22:48:45.714086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.710617    4837 cpu_manager.go:414] "containerMap: removing container" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy"
Nov 01 22:48:45.714086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.710649    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy"
Nov 01 22:48:45.714086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.710752    4837 cpu_manager.go:414] "containerMap: removing container" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy"
Nov 01 22:48:45.714086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.710760    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy"
Nov 01 22:48:45.714086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.710925    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy"
Nov 01 22:48:45.714086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.710941    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="600d0c78-8fc8-4f44-9b93-96951093c210" containerName="istio-proxy"
Nov 01 22:48:45.723071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.717613    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:45.756292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.756241    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"]
Nov 01 22:48:45.763746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.763317    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d57de51-eda3-4e97-b477-4e0bfd45e553-plugins-config-volume\") pod \"payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r\" (UID: \"3d57de51-eda3-4e97-b477-4e0bfd45e553\") " pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:45.763746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.763460    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jtw\" (UniqueName: \"kubernetes.io/projected/3d57de51-eda3-4e97-b477-4e0bfd45e553-kube-api-access-r4jtw\") pod \"payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r\" (UID: \"3d57de51-eda3-4e97-b477-4e0bfd45e553\") " pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:45.793585 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.793321    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-68a7f708\""
Nov 01 22:48:45.793585 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.793481    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-68a7f708-dockercfg-vqfgh\""
Nov 01 22:48:45.914309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.868953    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d57de51-eda3-4e97-b477-4e0bfd45e553-plugins-config-volume\") pod \"payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r\" (UID: \"3d57de51-eda3-4e97-b477-4e0bfd45e553\") " pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:45.914309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.869106    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jtw\" (UniqueName: \"kubernetes.io/projected/3d57de51-eda3-4e97-b477-4e0bfd45e553-kube-api-access-r4jtw\") pod \"payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r\" (UID: \"3d57de51-eda3-4e97-b477-4e0bfd45e553\") " pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:45.962203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.931133    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d57de51-eda3-4e97-b477-4e0bfd45e553-plugins-config-volume\") pod \"payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r\" (UID: \"3d57de51-eda3-4e97-b477-4e0bfd45e553\") " pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:45.986337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:45.985702    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"]
Nov 01 22:48:46.033526 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.025828    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jtw\" (UniqueName: \"kubernetes.io/projected/3d57de51-eda3-4e97-b477-4e0bfd45e553-kube-api-access-r4jtw\") pod \"payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r\" (UID: \"3d57de51-eda3-4e97-b477-4e0bfd45e553\") " pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:46.039742 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.038637    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:46.043856 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.043825    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"]
Nov 01 22:48:46.101319 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.101029    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:46.179072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.176731    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dce20b5-7772-4154-b4e6-70b379c1e439-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b\" (UID: \"8dce20b5-7772-4154-b4e6-70b379c1e439\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:46.179589 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.179534    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrk7j\" (UniqueName: \"kubernetes.io/projected/8dce20b5-7772-4154-b4e6-70b379c1e439-kube-api-access-nrk7j\") pod \"payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b\" (UID: \"8dce20b5-7772-4154-b4e6-70b379c1e439\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:46.281697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.281656    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrk7j\" (UniqueName: \"kubernetes.io/projected/8dce20b5-7772-4154-b4e6-70b379c1e439-kube-api-access-nrk7j\") pod \"payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b\" (UID: \"8dce20b5-7772-4154-b4e6-70b379c1e439\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:46.281903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.281884    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dce20b5-7772-4154-b4e6-70b379c1e439-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b\" (UID: \"8dce20b5-7772-4154-b4e6-70b379c1e439\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:46.282817 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.282795    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dce20b5-7772-4154-b4e6-70b379c1e439-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b\" (UID: \"8dce20b5-7772-4154-b4e6-70b379c1e439\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:46.336338 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.336308    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrk7j\" (UniqueName: \"kubernetes.io/projected/8dce20b5-7772-4154-b4e6-70b379c1e439-kube-api-access-nrk7j\") pod \"payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b\" (UID: \"8dce20b5-7772-4154-b4e6-70b379c1e439\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:46.520196 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.520108    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:46.559543 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:48:46.559517    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d57de51_eda3_4e97_b477_4e0bfd45e553.slice/crio-81ea7b66cf4ef1f0665801a19caafe5e91e5d3938eceb88537021ce4f4a3a346 WatchSource:0}: Error finding container 81ea7b66cf4ef1f0665801a19caafe5e91e5d3938eceb88537021ce4f4a3a346: Status 404 returned error can't find the container with id 81ea7b66cf4ef1f0665801a19caafe5e91e5d3938eceb88537021ce4f4a3a346
Nov 01 22:48:46.559679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.559660    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"]
Nov 01 22:48:46.991817 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:46.991790    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"]
Nov 01 22:48:47.192651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.192628    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:48:47.251497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251465    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-certs\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.251642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251517    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23ce78c-1db4-46e2-b04c-31c56f139dca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23ce78c-1db4-46e2-b04c-31c56f139dca-istiod-ca-cert\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.251642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251574    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-credential-socket\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.251642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251599    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-data\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.251879 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251645    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-kube-api-access-d7xcz\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-kube-api-access-d7xcz\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.251879 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251706    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-token\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-token\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.251879 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251761    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-certs" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:47.252031 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251907    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-credential-socket" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:47.252031 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.251980    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-envoy\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.252031 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.252004    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-data" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:47.252224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.252093    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23ce78c-1db4-46e2-b04c-31c56f139dca-istiod-ca-cert" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:47.252224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.252134    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-socket\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.252329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.252260    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-podinfo\") pod \"e23ce78c-1db4-46e2-b04c-31c56f139dca\" (UID: \"e23ce78c-1db4-46e2-b04c-31c56f139dca\") "
Nov 01 22:48:47.252384 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.252346    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-socket" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:47.253092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.253068    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.253224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.253096    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23ce78c-1db4-46e2-b04c-31c56f139dca-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.253224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.253113    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.253224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.253127    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.253224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.253141    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.254233 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.254210    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-token" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:47.254418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.254399    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-podinfo" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:48:47.254590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.254565    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-envoy" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:47.254590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.254577    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-kube-api-access-d7xcz" pod "e23ce78c-1db4-46e2-b04c-31c56f139dca" (UID: "e23ce78c-1db4-46e2-b04c-31c56f139dca"). InnerVolumeSpecName "kube-api-access-d7xcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:47.368067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.365439    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.368067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.365476    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.368067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.365495    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7xcz\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-kube-api-access-d7xcz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.368067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.365512    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23ce78c-1db4-46e2-b04c-31c56f139dca-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:47.568355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.567911    4837 generic.go:356] "Generic (PLEG): container finished" podID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerID="8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e" exitCode=0
Nov 01 22:48:47.568355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.567975    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"
Nov 01 22:48:47.568355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.568034    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" event={"ID":"e23ce78c-1db4-46e2-b04c-31c56f139dca","Type":"ContainerDied","Data":"8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e"}
Nov 01 22:48:47.568355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.568128    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6" event={"ID":"e23ce78c-1db4-46e2-b04c-31c56f139dca","Type":"ContainerDied","Data":"131112af802209d807e3fff2090660b8c3db551394cb89836618181c4334629e"}
Nov 01 22:48:47.568355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.568158    4837 scope.go:122] "RemoveContainer" containerID="8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e"
Nov 01 22:48:47.572498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.571793    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b" event={"ID":"8dce20b5-7772-4154-b4e6-70b379c1e439","Type":"ContainerStarted","Data":"fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1"}
Nov 01 22:48:47.572498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.571853    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b" event={"ID":"8dce20b5-7772-4154-b4e6-70b379c1e439","Type":"ContainerStarted","Data":"81366c6a51506308335ea47d32875d149bed3106198f20c8cee6a400ae0711f6"}
Nov 01 22:48:47.572498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.572439    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:47.575225 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.575177    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r" event={"ID":"3d57de51-eda3-4e97-b477-4e0bfd45e553","Type":"ContainerStarted","Data":"3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc"}
Nov 01 22:48:47.575412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.575392    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r" event={"ID":"3d57de51-eda3-4e97-b477-4e0bfd45e553","Type":"ContainerStarted","Data":"81ea7b66cf4ef1f0665801a19caafe5e91e5d3938eceb88537021ce4f4a3a346"}
Nov 01 22:48:47.575736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.575720    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:47.587767 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.587436    4837 scope.go:122] "RemoveContainer" containerID="077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f"
Nov 01 22:48:47.719313 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.719291    4837 scope.go:122] "RemoveContainer" containerID="8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e"
Nov 01 22:48:47.719817 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:47.719793    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e\": container with ID starting with 8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e not found: ID does not exist" containerID="8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e"
Nov 01 22:48:47.719920 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.719827    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e"} err="failed to get container status \"8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e\": rpc error: code = NotFound desc = could not find container \"8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e\": container with ID starting with 8a381f27324ef4ac6f808906af8fb5c0213704bdbdb3ac96a1427c514b9d394e not found: ID does not exist"
Nov 01 22:48:47.719920 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.719850    4837 scope.go:122] "RemoveContainer" containerID="077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f"
Nov 01 22:48:47.720184 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:47.720164    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f\": container with ID starting with 077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f not found: ID does not exist" containerID="077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f"
Nov 01 22:48:47.720246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.720190    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f"} err="failed to get container status \"077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f\": rpc error: code = NotFound desc = could not find container \"077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f\": container with ID starting with 077f7070bc5446732135d2ae65378c1e3b3495fbbf20f3741a4f8628d69b916f not found: ID does not exist"
Nov 01 22:48:47.764590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.760302    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r" podStartSLOduration=2.760283562 podStartE2EDuration="2.760283562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:47.752961157 +0000 UTC m=+1509.342160218" watchObservedRunningTime="2026-11-01 22:48:47.760283562 +0000 UTC m=+1509.349482626"
Nov 01 22:48:47.981074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:47.979708    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b" podStartSLOduration=2.9796871400000002 podStartE2EDuration="2.97968714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:47.893600748 +0000 UTC m=+1509.482799812" watchObservedRunningTime="2026-11-01 22:48:47.97968714 +0000 UTC m=+1509.568886202"
Nov 01 22:48:48.054536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.054497    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"]
Nov 01 22:48:48.120075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.115319    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b50b3403-openshift-default-68c44d4799-59lp6"]
Nov 01 22:48:48.532391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.532361    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" path="/var/lib/kubelet/pods/e23ce78c-1db4-46e2-b04c-31c56f139dca/volumes"
Nov 01 22:48:48.713089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.710547    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"]
Nov 01 22:48:48.713369 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.713101    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy"
Nov 01 22:48:48.713369 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.713136    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy"
Nov 01 22:48:48.713369 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.713154    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy"
Nov 01 22:48:48.713369 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.713162    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy"
Nov 01 22:48:48.713543 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.713464    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy"
Nov 01 22:48:48.713543 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.713529    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23ce78c-1db4-46e2-b04c-31c56f139dca" containerName="istio-proxy"
Nov 01 22:48:48.737068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.735569    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.832118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.822304    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-945efd02-openshift-default-dockercfg-fl45l\""
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843221    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843292    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdsnz\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-kube-api-access-gdsnz\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843435    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843568    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843623    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f09d829-20e3-4dee-9a0d-005e944488a9-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843662    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843686    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843711    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.843786    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.851064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.849257    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"]
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947169    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947336    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947393    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f09d829-20e3-4dee-9a0d-005e944488a9-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947431    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947452    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947507    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947579    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947742    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.947781    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdsnz\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-kube-api-access-gdsnz\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.948602    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.948826    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.949524    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f09d829-20e3-4dee-9a0d-005e944488a9-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:48.953067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:48.949787    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:49.044335 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:49.044256    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:49.150416 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:48:49.150329    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:48:49.204440 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:49.184706    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:49.229600 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:49.229531    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdsnz\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-kube-api-access-gdsnz\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:49.305329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:49.283167    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:49.305329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:49.283694    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:49.354066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:49.353431    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:50.338064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:50.304136    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:50.446401 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:50.445510    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:50.640004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:50.639963    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:50.640267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:50.640243    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:50.640813 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:50.640788    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:48:50.697062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:50.696205    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:50.964063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:50.962633    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"]
Nov 01 22:48:51.038507 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.030164    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" event={"ID":"2f09d829-20e3-4dee-9a0d-005e944488a9","Type":"ContainerStarted","Data":"43aa79585de921d3a4b318c271bbd7de63eac07474b6e7ea654363545ad807ca"}
Nov 01 22:48:51.306192 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.306132    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:51.306754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.306408    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:51.396068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.391367    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:48:51.433644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.433600    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:51.457065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.434178    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:48:51.458570 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.458237    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:51.463059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.459021    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:48:51.516969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.516884    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:48:51.561070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.560428    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:51.614594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.614563    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:48:51.621182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.621145    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:48:51.822396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.820840    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:51.830301 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.830121    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:48:51.848394 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:51.830487    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:48:52.077303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:52.077210    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" event={"ID":"2f09d829-20e3-4dee-9a0d-005e944488a9","Type":"ContainerStarted","Data":"5c0869dcac0aed0c435b2788436e9b602fadfc8c393a37fc7f8a0cfa1240a781"}
Nov 01 22:48:52.256145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:52.255514    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:48:52.284903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:52.277768    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" podStartSLOduration=4.277750491 podStartE2EDuration="4.277750491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:52.27734725 +0000 UTC m=+1513.866546313" watchObservedRunningTime="2026-11-01 22:48:52.277750491 +0000 UTC m=+1513.866949554"
Nov 01 22:48:52.365232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:52.363439    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:52.456091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:52.455364    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:52.521078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:52.509638    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:48:53.086110 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:53.086077    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:53.087058 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:53.087027    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:48:53.498167 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:53.498137    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:48:54.882560 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:54.882526    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"]
Nov 01 22:48:54.887701 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:54.887681    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:54.941357 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:54.941331    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"]
Nov 01 22:48:54.982324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:54.982292    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:48:54.982454 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:54.982344    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:48:55.015383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.015356    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.015534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.015400    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.015904 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.015879    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tznj\" (UniqueName: \"kubernetes.io/projected/250422a9-1111-43d5-9683-85cfa42c9f50-kube-api-access-9tznj\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.117603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.117575    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tznj\" (UniqueName: \"kubernetes.io/projected/250422a9-1111-43d5-9683-85cfa42c9f50-kube-api-access-9tznj\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.117752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.117632    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.117752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.117658    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.118573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.118550    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.119969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.119947    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.174852 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.174831    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tznj\" (UniqueName: \"kubernetes.io/projected/250422a9-1111-43d5-9683-85cfa42c9f50-kube-api-access-9tznj\") pod \"maas-api-revoke-sub-74c368b5-8xkbj\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.202216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.202197    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:55.584455 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:48:55.584423    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250422a9_1111_43d5_9683_85cfa42c9f50.slice/crio-7b423d6ef5f29a0004df5abcb33e0831d84257f09a5b7e2641428c43cc744825 WatchSource:0}: Error finding container 7b423d6ef5f29a0004df5abcb33e0831d84257f09a5b7e2641428c43cc744825: Status 404 returned error can't find the container with id 7b423d6ef5f29a0004df5abcb33e0831d84257f09a5b7e2641428c43cc744825
Nov 01 22:48:55.585234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:55.585194    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"]
Nov 01 22:48:56.111710 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:56.111629    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj" event={"ID":"250422a9-1111-43d5-9683-85cfa42c9f50","Type":"ContainerStarted","Data":"ea4429881ec83f97e274a9a8b739f8e6160e7c9c0a2e949c2c30d426cccb9bff"}
Nov 01 22:48:56.111710 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:56.111677    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj" event={"ID":"250422a9-1111-43d5-9683-85cfa42c9f50","Type":"ContainerStarted","Data":"7b423d6ef5f29a0004df5abcb33e0831d84257f09a5b7e2641428c43cc744825"}
Nov 01 22:48:56.231863 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:56.231033    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj" podStartSLOduration=2.2310182259999998 podStartE2EDuration="2.231018226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:56.228138038 +0000 UTC m=+1517.817337099" watchObservedRunningTime="2026-11-01 22:48:56.231018226 +0000 UTC m=+1517.820217325"
Nov 01 22:48:57.120424 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:57.120393    4837 generic.go:356] "Generic (PLEG): container finished" podID="250422a9-1111-43d5-9683-85cfa42c9f50" containerID="ea4429881ec83f97e274a9a8b739f8e6160e7c9c0a2e949c2c30d426cccb9bff" exitCode=0
Nov 01 22:48:57.120894 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:57.120474    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj" event={"ID":"250422a9-1111-43d5-9683-85cfa42c9f50","Type":"ContainerDied","Data":"ea4429881ec83f97e274a9a8b739f8e6160e7c9c0a2e949c2c30d426cccb9bff"}
Nov 01 22:48:58.303463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.303441    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:58.366351 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.366323    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-xks-ca\") pod \"250422a9-1111-43d5-9683-85cfa42c9f50\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") "
Nov 01 22:48:58.366596 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.366579    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-openshift-ca\") pod \"250422a9-1111-43d5-9683-85cfa42c9f50\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") "
Nov 01 22:48:58.366682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.366637    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250422a9-1111-43d5-9683-85cfa42c9f50-kube-api-access-9tznj\" (UniqueName: \"kubernetes.io/projected/250422a9-1111-43d5-9683-85cfa42c9f50-kube-api-access-9tznj\") pod \"250422a9-1111-43d5-9683-85cfa42c9f50\" (UID: \"250422a9-1111-43d5-9683-85cfa42c9f50\") "
Nov 01 22:48:58.366869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.366849    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-openshift-ca" pod "250422a9-1111-43d5-9683-85cfa42c9f50" (UID: "250422a9-1111-43d5-9683-85cfa42c9f50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:58.367301 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.367280    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:58.368360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.368340    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-xks-ca" pod "250422a9-1111-43d5-9683-85cfa42c9f50" (UID: "250422a9-1111-43d5-9683-85cfa42c9f50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:58.368617 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.368596    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250422a9-1111-43d5-9683-85cfa42c9f50-kube-api-access-9tznj" pod "250422a9-1111-43d5-9683-85cfa42c9f50" (UID: "250422a9-1111-43d5-9683-85cfa42c9f50"). InnerVolumeSpecName "kube-api-access-9tznj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:58.469198 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.469167    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tznj\" (UniqueName: \"kubernetes.io/projected/250422a9-1111-43d5-9683-85cfa42c9f50-kube-api-access-9tznj\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:58.469198 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.469198    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250422a9-1111-43d5-9683-85cfa42c9f50-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:58.709717 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.709361    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:48:58.761114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:58.715177    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:48:59.086863 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.086776    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"]
Nov 01 22:48:59.094700 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.094666    4837 cpu_manager.go:414] "containerMap: removing container" podUID="250422a9-1111-43d5-9683-85cfa42c9f50" containerName="revoke-keys"
Nov 01 22:48:59.096112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.096086    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250422a9-1111-43d5-9683-85cfa42c9f50" containerName="revoke-keys"
Nov 01 22:48:59.096719 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.096699    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="250422a9-1111-43d5-9683-85cfa42c9f50" containerName="revoke-keys"
Nov 01 22:48:59.107137 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.107113    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:48:59.175070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.168556    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"]
Nov 01 22:48:59.211747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.208963    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-945efd02-dockercfg-568fm\""
Nov 01 22:48:59.211747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.209105    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-945efd02\""
Nov 01 22:48:59.378970 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.378112    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk9p2\" (UniqueName: \"kubernetes.io/projected/fa512302-8ce9-4aec-95ce-1249b284107e-kube-api-access-mk9p2\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579\" (UID: \"fa512302-8ce9-4aec-95ce-1249b284107e\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:48:59.378970 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.378196    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa512302-8ce9-4aec-95ce-1249b284107e-plugins-config-volume\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579\" (UID: \"fa512302-8ce9-4aec-95ce-1249b284107e\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:48:59.417680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.417644    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj" event={"ID":"250422a9-1111-43d5-9683-85cfa42c9f50","Type":"ContainerDied","Data":"7b423d6ef5f29a0004df5abcb33e0831d84257f09a5b7e2641428c43cc744825"}
Nov 01 22:48:59.417929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.417911    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b423d6ef5f29a0004df5abcb33e0831d84257f09a5b7e2641428c43cc744825"
Nov 01 22:48:59.418109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.418096    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"
Nov 01 22:48:59.480138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.479294    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk9p2\" (UniqueName: \"kubernetes.io/projected/fa512302-8ce9-4aec-95ce-1249b284107e-kube-api-access-mk9p2\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579\" (UID: \"fa512302-8ce9-4aec-95ce-1249b284107e\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:48:59.480138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.479363    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa512302-8ce9-4aec-95ce-1249b284107e-plugins-config-volume\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579\" (UID: \"fa512302-8ce9-4aec-95ce-1249b284107e\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:48:59.494038 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.480966    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa512302-8ce9-4aec-95ce-1249b284107e-plugins-config-volume\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579\" (UID: \"fa512302-8ce9-4aec-95ce-1249b284107e\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:48:59.643159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.628829    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk9p2\" (UniqueName: \"kubernetes.io/projected/fa512302-8ce9-4aec-95ce-1249b284107e-kube-api-access-mk9p2\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579\" (UID: \"fa512302-8ce9-4aec-95ce-1249b284107e\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:48:59.860168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:48:59.860141    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:49:00.235631 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:49:00.235596    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa512302_8ce9_4aec_95ce_1249b284107e.slice/crio-556c0e3f46e9b8e801a687b6db1e9a25b6d36a6525ed535dbf763bc819334fc8 WatchSource:0}: Error finding container 556c0e3f46e9b8e801a687b6db1e9a25b6d36a6525ed535dbf763bc819334fc8: Status 404 returned error can't find the container with id 556c0e3f46e9b8e801a687b6db1e9a25b6d36a6525ed535dbf763bc819334fc8
Nov 01 22:49:00.237421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:00.237395    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"]
Nov 01 22:49:00.427372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:00.427341    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579" event={"ID":"fa512302-8ce9-4aec-95ce-1249b284107e","Type":"ContainerStarted","Data":"9f2fd08824e9aed4ea8eb7c705fed96ff7652d6b7c3a00033dc0bb492309610a"}
Nov 01 22:49:00.427734 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:00.427396    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579" event={"ID":"fa512302-8ce9-4aec-95ce-1249b284107e","Type":"ContainerStarted","Data":"556c0e3f46e9b8e801a687b6db1e9a25b6d36a6525ed535dbf763bc819334fc8"}
Nov 01 22:49:00.427734 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:00.427434    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:49:00.541616 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:00.541534    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579" podStartSLOduration=2.541521416 podStartE2EDuration="2.541521416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:00.536423883 +0000 UTC m=+1522.125622944" watchObservedRunningTime="2026-11-01 22:49:00.541521416 +0000 UTC m=+1522.130720817"
Nov 01 22:49:11.438130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:11.438094    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:49:17.180531 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.180496    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"]
Nov 01 22:49:17.186933 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.186914    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.247898 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.247873    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d04c71aa-serving-cert\""
Nov 01 22:49:17.294165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.294127    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"]
Nov 01 22:49:17.316272 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.316242    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tpvs\" (UniqueName: \"kubernetes.io/projected/f475bb62-cd9f-43e0-b045-f50944e6d69e-kube-api-access-9tpvs\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.316452 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.316318    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-maas-api-tls\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.316622 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.316596    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f475bb62-cd9f-43e0-b045-f50944e6d69e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.316687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.316647    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-metrics-certs\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.429129 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.428973    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tpvs\" (UniqueName: \"kubernetes.io/projected/f475bb62-cd9f-43e0-b045-f50944e6d69e-kube-api-access-9tpvs\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.429772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.429744    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-maas-api-tls\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.430165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.430146    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f475bb62-cd9f-43e0-b045-f50944e6d69e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.433280 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.431421    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f475bb62-cd9f-43e0-b045-f50944e6d69e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.433425 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.433408    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-metrics-certs\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.471736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.471703    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-metrics-certs\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.475672 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.474032    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-maas-api-tls\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.492058 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.492006    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tpvs\" (UniqueName: \"kubernetes.io/projected/f475bb62-cd9f-43e0-b045-f50944e6d69e-kube-api-access-9tpvs\") pod \"maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:17.500679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:17.500653    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:18.088293 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:49:18.088034    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf475bb62_cd9f_43e0_b045_f50944e6d69e.slice/crio-2b3b35d47ade3abad2510bc0026f0cf27bc68b9298cbf65b39e0f97221148a0e WatchSource:0}: Error finding container 2b3b35d47ade3abad2510bc0026f0cf27bc68b9298cbf65b39e0f97221148a0e: Status 404 returned error can't find the container with id 2b3b35d47ade3abad2510bc0026f0cf27bc68b9298cbf65b39e0f97221148a0e
Nov 01 22:49:18.091200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:18.091174    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"]
Nov 01 22:49:18.746038 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:18.746004    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76" event={"ID":"f475bb62-cd9f-43e0-b045-f50944e6d69e","Type":"ContainerStarted","Data":"2b3b35d47ade3abad2510bc0026f0cf27bc68b9298cbf65b39e0f97221148a0e"}
Nov 01 22:49:18.813788 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:49:18.813756    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:49:19.756435 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:19.756401    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76" event={"ID":"f475bb62-cd9f-43e0-b045-f50944e6d69e","Type":"ContainerStarted","Data":"4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c"}
Nov 01 22:49:19.756885 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:19.756462    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:19.869801 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:19.869756    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76" podStartSLOduration=2.00445662 podStartE2EDuration="2.869741581s" totalImagesPullingTime="865.284961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:19.869248439 +0000 UTC m=+1541.458447502" watchObservedRunningTime="2026-11-01 22:49:19.869741581 +0000 UTC m=+1541.458940641"
Nov 01 22:49:20.056114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:20.055328    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"]
Nov 01 22:49:20.131132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:20.111680    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ceef79c-lvdpz"]
Nov 01 22:49:20.529963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:20.529927    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839b7f71-5fc5-450c-91f5-3bacd72e8eac" path="/var/lib/kubelet/pods/839b7f71-5fc5-450c-91f5-3bacd72e8eac/volumes"
Nov 01 22:49:24.085911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:24.085875    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:49:25.772979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:25.772944    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:49:27.664618 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:27.664560    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:49:27.665363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:27.665327    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:49:27.799241 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:27.799199    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:49:27.873467 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:27.873329    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Nov 01 22:49:28.291068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:28.286209    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:49:28.291068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:28.286548    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:49:28.450175 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:28.450118    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:49:28.473905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:28.450566    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 22:49:28.540335 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:28.540290    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:49:28.547560 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:28.547468    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:49:28.571222 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:28.571185    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:49:28.625072 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:28.623668    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:49:34.778526 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.778492    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"]
Nov 01 22:49:34.784455 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.784434    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.842704 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.836482    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-de3d6ade-openshift-default-dockercfg-fkbf7\""
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.888395    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-socket\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.888466    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-credential-socket\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.888493    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istiod-ca-cert\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.888543    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-envoy\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.888664    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-certs\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.888811    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-podinfo\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.888887    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfs2\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-kube-api-access-6xfs2\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.888938    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-data\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:34.889092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:34.889031    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-token\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.133709 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.133634    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-podinfo\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.134192 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.134160    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfs2\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-kube-api-access-6xfs2\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.134314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.134221    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-data\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.134314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.134299    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-token\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.135078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.134357    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-socket\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.135078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.134395    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-credential-socket\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.135078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.134414    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istiod-ca-cert\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.135078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.134456    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-envoy\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.135078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.134554    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-certs\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.152362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.135073    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-certs\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.152362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.135375    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-socket\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.152362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.135645    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-credential-socket\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.152362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.136252    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istiod-ca-cert\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.152362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.141824    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-data\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.152362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.148862    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"]
Nov 01 22:49:35.179388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.170397    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-envoy\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.214243 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.214148    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-token\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.240762 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.240692    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-podinfo\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.271472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.267232    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfs2\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-kube-api-access-6xfs2\") pod \"e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") " pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.410109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.402019    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:35.871421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.871394    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"]
Nov 01 22:49:35.874976 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:49:35.874948    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f1a6f7_57be_4ca6_86e0_a45714f3f7b7.slice/crio-f77e913fbf3039030fcc6e2bf4dbbc50f61dcf1c4d4b98b2cb84916c417902c6 WatchSource:0}: Error finding container f77e913fbf3039030fcc6e2bf4dbbc50f61dcf1c4d4b98b2cb84916c417902c6: Status 404 returned error can't find the container with id f77e913fbf3039030fcc6e2bf4dbbc50f61dcf1c4d4b98b2cb84916c417902c6
Nov 01 22:49:35.877739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.877707    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:49:35.877840 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.877774    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:49:35.877840 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:35.877820    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:49:36.374076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:36.370595    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" event={"ID":"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7","Type":"ContainerStarted","Data":"055aaae705f7618bdd13077ab1766012f0de79cf7f423f65bfc26c20d02d2677"}
Nov 01 22:49:36.374076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:36.370666    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" event={"ID":"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7","Type":"ContainerStarted","Data":"f77e913fbf3039030fcc6e2bf4dbbc50f61dcf1c4d4b98b2cb84916c417902c6"}
Nov 01 22:49:36.425066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:36.413397    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:36.425066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:36.413523    4837 patch_prober.go:29] interesting pod/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 01 22:49:36.425066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:36.413585    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 01 22:49:36.592160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:36.561212    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podStartSLOduration=2.561189304 podStartE2EDuration="2.561189304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:36.540347712 +0000 UTC m=+1558.129546773" watchObservedRunningTime="2026-11-01 22:49:36.561189304 +0000 UTC m=+1558.150388371"
Nov 01 22:49:37.111203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:37.111146    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"]
Nov 01 22:49:37.318736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:37.305805    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b9ee65-b5jxz"]
Nov 01 22:49:37.584375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:37.584344    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:38.529926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:38.529898    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b962a16e-cb7b-4ed6-99ab-999944397b9b" path="/var/lib/kubelet/pods/b962a16e-cb7b-4ed6-99ab-999944397b9b/volumes"
Nov 01 22:49:38.623573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:38.623460    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:38.633065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:38.624908    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:49:41.052894 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:41.052856    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"]
Nov 01 22:49:41.098334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:41.098304    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91a3ead-lrxwt"]
Nov 01 22:49:42.527860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:42.527823    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1c7b45-8473-43da-860d-8057a1413da5" path="/var/lib/kubelet/pods/7d1c7b45-8473-43da-860d-8057a1413da5/volumes"
Nov 01 22:49:48.754102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:48.754000    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"]
Nov 01 22:49:48.763199 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:48.763171    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:48.815680 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:48.815651    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"]
Nov 01 22:49:48.817609 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:49:48.817581    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:49:48.854872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:48.854841    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-945efd02-serving-cert\""
Nov 01 22:49:48.915636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:48.915608    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:48.915871 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:48.915666    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-metrics-certs\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:48.915871 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:48.915840    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxw4\" (UniqueName: \"kubernetes.io/projected/59837a37-e54d-4458-a7a3-15631f572036-kube-api-access-wsxw4\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:48.916066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:48.916029    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59837a37-e54d-4458-a7a3-15631f572036-openshift-service-ca\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.017707 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.017638    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59837a37-e54d-4458-a7a3-15631f572036-openshift-service-ca\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.017849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.017826    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.017891 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.017857    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-metrics-certs\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.017940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.017891    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxw4\" (UniqueName: \"kubernetes.io/projected/59837a37-e54d-4458-a7a3-15631f572036-kube-api-access-wsxw4\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.018008 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:49:49.017987    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-serving-cert: secret "maas-api-e2e-mt-945efd02-serving-cert" not found
Nov 01 22:49:49.018125 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:49:49.018111    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls podName:59837a37-e54d-4458-a7a3-15631f572036 nodeName:}" failed. No retries permitted until 2026-11-01 22:49:49.518090923 +0000 UTC m=+1571.107289976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls") pod "maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl" (UID: "59837a37-e54d-4458-a7a3-15631f572036") : secret "maas-api-e2e-mt-945efd02-serving-cert" not found
Nov 01 22:49:49.018285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.018267    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59837a37-e54d-4458-a7a3-15631f572036-openshift-service-ca\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.020292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.020274    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-metrics-certs\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.141492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.126148    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxw4\" (UniqueName: \"kubernetes.io/projected/59837a37-e54d-4458-a7a3-15631f572036-kube-api-access-wsxw4\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.558867 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.558834    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.561649 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.561620    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:49.677980 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:49.677920    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:50.053811 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:49:50.053779    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59837a37_e54d_4458_a7a3_15631f572036.slice/crio-647cbcc8be6abfb09f0a706abfbab87624ef28b2a9fca43ad05dbb844854694b WatchSource:0}: Error finding container 647cbcc8be6abfb09f0a706abfbab87624ef28b2a9fca43ad05dbb844854694b: Status 404 returned error can't find the container with id 647cbcc8be6abfb09f0a706abfbab87624ef28b2a9fca43ad05dbb844854694b
Nov 01 22:49:50.054906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:50.054885    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"]
Nov 01 22:49:50.909221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:50.909186    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl" event={"ID":"59837a37-e54d-4458-a7a3-15631f572036","Type":"ContainerStarted","Data":"647cbcc8be6abfb09f0a706abfbab87624ef28b2a9fca43ad05dbb844854694b"}
Nov 01 22:49:51.919349 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:51.919310    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl" event={"ID":"59837a37-e54d-4458-a7a3-15631f572036","Type":"ContainerStarted","Data":"126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746"}
Nov 01 22:49:51.919789 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:51.919398    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:57.952225 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:57.952193    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:49:58.084283 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:49:58.084217    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl" podStartSLOduration=9.199247247 podStartE2EDuration="10.08419526s" totalImagesPullingTime="884.948013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:52.03651349 +0000 UTC m=+1573.625712554" watchObservedRunningTime="2026-11-01 22:49:58.08419526 +0000 UTC m=+1579.673394322"
Nov 01 22:50:04.950214 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:04.950166    4837 patch_prober.go:29] interesting pod/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:50:04.950688 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:04.950264    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:50:05.060804 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:05.060767    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"]
Nov 01 22:50:05.104339 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:05.104308    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ac09baa-wc4sr"]
Nov 01 22:50:06.244784 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.244756    4837 generic.go:356] "Generic (PLEG): container finished" podID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerID="8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62" exitCode=0
Nov 01 22:50:06.245281 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.244811    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" event={"ID":"7119d22b-44f1-4c43-8a7b-c1dc954edaab","Type":"ContainerDied","Data":"8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62"}
Nov 01 22:50:06.245349 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.245295    4837 scope.go:122] "RemoveContainer" containerID="8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62"
Nov 01 22:50:06.246925 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.246904    4837 generic.go:356] "Generic (PLEG): container finished" podID="e7646a63-328d-4629-bd52-0f0f50486d82" containerID="d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8" exitCode=0
Nov 01 22:50:06.247031 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.246988    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" event={"ID":"e7646a63-328d-4629-bd52-0f0f50486d82","Type":"ContainerDied","Data":"d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8"}
Nov 01 22:50:06.247121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.247091    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:06.247179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.247163    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:06.247220 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.247207    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:06.247379 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.247362    4837 scope.go:122] "RemoveContainer" containerID="d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8"
Nov 01 22:50:06.248819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.248793    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:06.248930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.248862    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:06.248930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.248894    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:06.535872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:06.535783    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323cb200-8abe-47de-820a-95d8c26e4fa2" path="/var/lib/kubelet/pods/323cb200-8abe-47de-820a-95d8c26e4fa2/volumes"
Nov 01 22:50:07.290788 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:07.289358    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" event={"ID":"7119d22b-44f1-4c43-8a7b-c1dc954edaab","Type":"ContainerStarted","Data":"f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9"}
Nov 01 22:50:07.291867 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:07.291840    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" event={"ID":"e7646a63-328d-4629-bd52-0f0f50486d82","Type":"ContainerStarted","Data":"b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93"}
Nov 01 22:50:07.626002 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:07.625916    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:50:07.626201 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:07.626158    4837 patch_prober.go:29] interesting pod/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:50:07.626275 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:07.626202    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 01 22:50:07.877132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:07.874901    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:50:07.877132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:07.875013    4837 patch_prober.go:29] interesting pod/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:50:07.877132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:07.875075    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:50:08.626062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:08.626018    4837 patch_prober.go:29] interesting pod/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:50:08.626537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:08.626090    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 01 22:50:08.876123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:08.869546    4837 patch_prober.go:29] interesting pod/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:50:08.876123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:08.869603    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:50:09.624913 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:09.624882    4837 patch_prober.go:29] interesting pod/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:50:09.625093 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:09.624943    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 01 22:50:09.883819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:09.883127    4837 patch_prober.go:29] interesting pod/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:50:09.883819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:09.883186    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:50:10.083492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:10.083454    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"]
Nov 01 22:50:10.088788 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:10.088759    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-791655fd-hxkjq"]
Nov 01 22:50:10.471184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:10.471142    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:50:10.528418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:10.528380    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908370d0-afcf-497c-b76f-889dc8ae01a3" path="/var/lib/kubelet/pods/908370d0-afcf-497c-b76f-889dc8ae01a3/volumes"
Nov 01 22:50:10.625262 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:10.625228    4837 patch_prober.go:29] interesting pod/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:50:10.625426 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:10.625272    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 01 22:50:10.871120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:10.870674    4837 patch_prober.go:29] interesting pod/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:50:10.871120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:10.870730    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:50:11.624829 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:11.624797    4837 patch_prober.go:29] interesting pod/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:50:11.625314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:11.624854    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 01 22:50:11.876117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:11.870364    4837 patch_prober.go:29] interesting pod/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:50:11.876117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:11.870426    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:50:12.628562 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:12.628530    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:50:12.629075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:12.629025    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:50:12.629794 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:12.629773    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:50:12.873936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:12.873908    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:50:12.874906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:12.874887    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:50:13.063269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:13.063237    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"]
Nov 01 22:50:13.069715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:13.069689    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d52b510-r4nqs"]
Nov 01 22:50:14.527853 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:14.527821    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05" path="/var/lib/kubelet/pods/1dbdba72-c4af-4d9f-95fd-fb3c68ec2d05/volumes"
Nov 01 22:50:17.072527 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:17.072496    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"]
Nov 01 22:50:17.118619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:17.118594    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc330a-9lkkb"]
Nov 01 22:50:18.527978 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:18.527942    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d980712-fd80-4bf0-98e3-1aa34de5b4f5" path="/var/lib/kubelet/pods/3d980712-fd80-4bf0-98e3-1aa34de5b4f5/volumes"
Nov 01 22:50:18.813957 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:50:18.813869    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:50:25.170951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:25.170909    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:50:32.067190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:32.067159    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"]
Nov 01 22:50:32.151117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:32.150568    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12dd1fb8-sjnqf"]
Nov 01 22:50:32.527746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:32.527709    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48c3035-122d-4c47-9310-fc188aab058b" path="/var/lib/kubelet/pods/f48c3035-122d-4c47-9310-fc188aab058b/volumes"
Nov 01 22:50:33.770655 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:33.770614    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"]
Nov 01 22:50:33.876838 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:33.876801    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"]
Nov 01 22:50:34.048549 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:34.048462    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"]
Nov 01 22:50:34.054657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:34.054629    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24113d11-4b4xl"]
Nov 01 22:50:34.654446 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:34.654406    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69654a5-8279-4298-b61d-8f56a47a4a79" path="/var/lib/kubelet/pods/a69654a5-8279-4298-b61d-8f56a47a4a79/volumes"
Nov 01 22:50:37.705888 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:37.705858    4837 generic.go:356] "Generic (PLEG): container finished" podID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerID="5c0869dcac0aed0c435b2788436e9b602fadfc8c393a37fc7f8a0cfa1240a781" exitCode=0
Nov 01 22:50:37.706369 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:37.705932    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" event={"ID":"2f09d829-20e3-4dee-9a0d-005e944488a9","Type":"ContainerDied","Data":"5c0869dcac0aed0c435b2788436e9b602fadfc8c393a37fc7f8a0cfa1240a781"}
Nov 01 22:50:37.706369 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:37.706337    4837 scope.go:122] "RemoveContainer" containerID="5c0869dcac0aed0c435b2788436e9b602fadfc8c393a37fc7f8a0cfa1240a781"
Nov 01 22:50:37.707890 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:37.707862    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:37.707950 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:37.707936    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:37.707997 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:37.707970    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:50:38.087022 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:38.086949    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:50:38.715837 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:38.715802    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" event={"ID":"2f09d829-20e3-4dee-9a0d-005e944488a9","Type":"ContainerStarted","Data":"d8cad0fb4fb3691853420f1d32c041b31fd1d983ea69ccb3c7737a60481d7540"}
Nov 01 22:50:38.771659 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:38.771621    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz" podUID="9f2952f9-61a3-4869-9da0-999c4caeea56" containerName="payload-processing" containerID="cri-o://37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee" gracePeriod=25
Nov 01 22:50:38.877842 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:38.877774    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt" podUID="1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f" containerName="payload-pre-processing" containerID="cri-o://8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae" gracePeriod=25
Nov 01 22:50:38.962409 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:38.962378    4837 scope.go:122] "RemoveContainer" containerID="4826b3ddf750a4906f81dc7a58cab20634244d4fb39c6c590e9d13ef4291d401"
Nov 01 22:50:38.997146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:38.996747    4837 scope.go:122] "RemoveContainer" containerID="c9ba98aca4d3c3c087583984df55f9efe43735a1262a9a6b1bffa396cea6efdc"
Nov 01 22:50:39.012445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.012395    4837 scope.go:122] "RemoveContainer" containerID="5ddb629febe024b61ccf8ea1ea97996f50224e45d0a4b00ba3f3603f4347c534"
Nov 01 22:50:39.129825 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.127701    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:50:39.267962 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.267935    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:50:39.285377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.285348    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2952f9-61a3-4869-9da0-999c4caeea56-kube-api-access-bbm4f\" (UniqueName: \"kubernetes.io/projected/9f2952f9-61a3-4869-9da0-999c4caeea56-kube-api-access-bbm4f\") pod \"9f2952f9-61a3-4869-9da0-999c4caeea56\" (UID: \"9f2952f9-61a3-4869-9da0-999c4caeea56\") "
Nov 01 22:50:39.285530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.285418    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2952f9-61a3-4869-9da0-999c4caeea56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2952f9-61a3-4869-9da0-999c4caeea56-plugins-config-volume\") pod \"9f2952f9-61a3-4869-9da0-999c4caeea56\" (UID: \"9f2952f9-61a3-4869-9da0-999c4caeea56\") "
Nov 01 22:50:39.285892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.285859    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2952f9-61a3-4869-9da0-999c4caeea56-plugins-config-volume" pod "9f2952f9-61a3-4869-9da0-999c4caeea56" (UID: "9f2952f9-61a3-4869-9da0-999c4caeea56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:50:39.286445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.286417    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2952f9-61a3-4869-9da0-999c4caeea56-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:39.287864 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.287838    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2952f9-61a3-4869-9da0-999c4caeea56-kube-api-access-bbm4f" pod "9f2952f9-61a3-4869-9da0-999c4caeea56" (UID: "9f2952f9-61a3-4869-9da0-999c4caeea56"). InnerVolumeSpecName "kube-api-access-bbm4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:50:39.355404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.355333    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:50:39.356214 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.356179    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:50:39.356346 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.356240    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:50:39.387852 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.387828    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-kube-api-access-rm54r\" (UniqueName: \"kubernetes.io/projected/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-kube-api-access-rm54r\") pod \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\" (UID: \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\") "
Nov 01 22:50:39.387997 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.387952    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-plugins-config-volume\") pod \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\" (UID: \"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f\") "
Nov 01 22:50:39.388359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.388338    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-plugins-config-volume" pod "1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f" (UID: "1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:50:39.388635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.388616    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbm4f\" (UniqueName: \"kubernetes.io/projected/9f2952f9-61a3-4869-9da0-999c4caeea56-kube-api-access-bbm4f\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:39.388635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.388632    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:39.390206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.390177    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-kube-api-access-rm54r" pod "1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f" (UID: "1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f"). InnerVolumeSpecName "kube-api-access-rm54r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:50:39.490284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.490212    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm54r\" (UniqueName: \"kubernetes.io/projected/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f-kube-api-access-rm54r\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:39.725593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.725562    4837 generic.go:356] "Generic (PLEG): container finished" podID="9f2952f9-61a3-4869-9da0-999c4caeea56" containerID="37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee" exitCode=0
Nov 01 22:50:39.726069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.725604    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"
Nov 01 22:50:39.726069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.725645    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz" event={"ID":"9f2952f9-61a3-4869-9da0-999c4caeea56","Type":"ContainerDied","Data":"37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee"}
Nov 01 22:50:39.726069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.725695    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz" event={"ID":"9f2952f9-61a3-4869-9da0-999c4caeea56","Type":"ContainerDied","Data":"393150eeef5db912056c0c1c5c1939154c0c1e19b6c548f0ef7e62f2da89b140"}
Nov 01 22:50:39.726069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.725724    4837 scope.go:122] "RemoveContainer" containerID="37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee"
Nov 01 22:50:39.727373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.727347    4837 generic.go:356] "Generic (PLEG): container finished" podID="1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f" containerID="8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae" exitCode=0
Nov 01 22:50:39.727488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.727395    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"
Nov 01 22:50:39.727488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.727422    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt" event={"ID":"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f","Type":"ContainerDied","Data":"8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae"}
Nov 01 22:50:39.727488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.727466    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt" event={"ID":"1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f","Type":"ContainerDied","Data":"1380fff35f22af35d00ac2d9c0f24585296ea94306d69b5cd14c23ceef64c554"}
Nov 01 22:50:39.764182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.764160    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"]
Nov 01 22:50:39.764987 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.764965    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9f2952f9-61a3-4869-9da0-999c4caeea56" containerName="payload-processing"
Nov 01 22:50:39.764987 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.764985    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2952f9-61a3-4869-9da0-999c4caeea56" containerName="payload-processing"
Nov 01 22:50:39.765179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.765030    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f" containerName="payload-pre-processing"
Nov 01 22:50:39.765179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.765036    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f" containerName="payload-pre-processing"
Nov 01 22:50:39.765179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.765179    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f" containerName="payload-pre-processing"
Nov 01 22:50:39.765353 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.765193    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2952f9-61a3-4869-9da0-999c4caeea56" containerName="payload-processing"
Nov 01 22:50:39.765907 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.765891    4837 scope.go:122] "RemoveContainer" containerID="37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee"
Nov 01 22:50:39.766374 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:50:39.766354    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee\": container with ID starting with 37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee not found: ID does not exist" containerID="37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee"
Nov 01 22:50:39.766450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.766380    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee"} err="failed to get container status \"37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee\": rpc error: code = NotFound desc = could not find container \"37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee\": container with ID starting with 37bd3af4289e42a0f81cfd5049b7b0b8553cbcbc051af807d41236bdd44aadee not found: ID does not exist"
Nov 01 22:50:39.766450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.766398    4837 scope.go:122] "RemoveContainer" containerID="8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae"
Nov 01 22:50:39.769143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.769126    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:39.782473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.779217    4837 scope.go:122] "RemoveContainer" containerID="8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae"
Nov 01 22:50:39.782473 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:50:39.779554    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae\": container with ID starting with 8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae not found: ID does not exist" containerID="8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae"
Nov 01 22:50:39.782473 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.779590    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae"} err="failed to get container status \"8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae\": rpc error: code = NotFound desc = could not find container \"8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae\": container with ID starting with 8a788f6c09a029ff1c649f0d51b87004285562132f430a02cb69bc07a6202eae not found: ID does not exist"
Nov 01 22:50:39.827085 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.823142    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-de3d6ade-serving-cert\""
Nov 01 22:50:39.827085 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.826676    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"]
Nov 01 22:50:39.896358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.896326    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-maas-api-tls\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:39.896584 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.896565    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29mmb\" (UniqueName: \"kubernetes.io/projected/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-kube-api-access-29mmb\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:39.896826 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.896794    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-metrics-certs\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:39.896965 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.896881    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-openshift-service-ca\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:39.899594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.899575    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"]
Nov 01 22:50:39.905772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:39.905751    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d04c71aa-f9fcb668c-zf5lz"]
Nov 01 22:50:40.037114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.014397    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-metrics-certs\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.037114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.014445    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-openshift-service-ca\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.037114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.014897    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-maas-api-tls\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.037114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.015240    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29mmb\" (UniqueName: \"kubernetes.io/projected/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-kube-api-access-29mmb\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.037114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.015849    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-openshift-service-ca\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.096065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.095841    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29mmb\" (UniqueName: \"kubernetes.io/projected/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-kube-api-access-29mmb\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.102063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.097710    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-maas-api-tls\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.122066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.114127    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-metrics-certs\") pod \"maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.200071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.193112    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"]
Nov 01 22:50:40.200071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.194873    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d04c71aa-698d874d87-4jmzt"]
Nov 01 22:50:40.359555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.359464    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:50:40.359850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.359818    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:50:40.386926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.386808    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:40.550068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.533839    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f" path="/var/lib/kubelet/pods/1c2f6a4f-c53d-45c4-a48d-6f6ff7ec3b7f/volumes"
Nov 01 22:50:40.550068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.534572    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2952f9-61a3-4869-9da0-999c4caeea56" path="/var/lib/kubelet/pods/9f2952f9-61a3-4869-9da0-999c4caeea56/volumes"
Nov 01 22:50:40.881471 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:50:40.881434    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb12256_dfbe_401b_91b5_ee1f9a2106cd.slice/crio-dbae5e37695dccfeb8c6861389e4c27af453c4195672a6dc4c65806607b8a791 WatchSource:0}: Error finding container dbae5e37695dccfeb8c6861389e4c27af453c4195672a6dc4c65806607b8a791: Status 404 returned error can't find the container with id dbae5e37695dccfeb8c6861389e4c27af453c4195672a6dc4c65806607b8a791
Nov 01 22:50:40.882140 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:40.882055    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"]
Nov 01 22:50:41.060412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:41.060376    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"]
Nov 01 22:50:41.064296 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:41.064273    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1152c744-rsnrh"]
Nov 01 22:50:41.355518 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:41.355479    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:50:41.355699 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:41.355586    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:50:41.770802 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:41.770769    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8" event={"ID":"bcb12256-dfbe-401b-91b5-ee1f9a2106cd","Type":"ContainerStarted","Data":"dbae5e37695dccfeb8c6861389e4c27af453c4195672a6dc4c65806607b8a791"}
Nov 01 22:50:42.355470 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:42.355430    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:50:42.355883 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:42.355497    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:50:42.527314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:42.527282    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e7028a-755b-435b-a3a7-338b93cd6132" path="/var/lib/kubelet/pods/35e7028a-755b-435b-a3a7-338b93cd6132/volumes"
Nov 01 22:50:42.781386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:42.781352    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8" event={"ID":"bcb12256-dfbe-401b-91b5-ee1f9a2106cd","Type":"ContainerStarted","Data":"44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713"}
Nov 01 22:50:42.781557 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:42.781444    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:42.945066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:42.928427    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8" podStartSLOduration=3.000854834 podStartE2EDuration="3.928405048s" totalImagesPullingTime="927.550214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:42.925379948 +0000 UTC m=+1624.514579009" watchObservedRunningTime="2026-11-01 22:50:42.928405048 +0000 UTC m=+1624.517604111"
Nov 01 22:50:43.368069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:43.367560    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:50:43.368069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:43.367636    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:50:44.360314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:44.360284    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:50:44.360724 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:44.360703    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:50:44.361564 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:44.361543    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 22:50:48.045560 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:48.045525    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"]
Nov 01 22:50:48.091403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:48.091370    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997e8a96-8pcht"]
Nov 01 22:50:48.527341 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:48.527311    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97581061-e864-4439-95d2-458e444b18cc" path="/var/lib/kubelet/pods/97581061-e864-4439-95d2-458e444b18cc/volumes"
Nov 01 22:50:48.794138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:48.794038    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:50:48.817246 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:50:48.817212    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:50:51.272674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:51.272635    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"]
Nov 01 22:50:51.377278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:51.377246    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"]
Nov 01 22:50:53.074749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:53.074713    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"]
Nov 01 22:50:53.166222 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:53.163341    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee0c0d-9dkkl"]
Nov 01 22:50:54.529500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:54.529468    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f" path="/var/lib/kubelet/pods/58a7b442-6fc7-4a6c-a1ba-d3a4c1a6a36f/volumes"
Nov 01 22:50:56.272998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.272942    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r" podUID="3d57de51-eda3-4e97-b477-4e0bfd45e553" containerName="payload-processing" containerID="cri-o://3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc" gracePeriod=25
Nov 01 22:50:56.378173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.378125    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b" podUID="8dce20b5-7772-4154-b4e6-70b379c1e439" containerName="payload-pre-processing" containerID="cri-o://fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1" gracePeriod=25
Nov 01 22:50:56.587939 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.587912    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:50:56.672763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.672728    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d57de51-eda3-4e97-b477-4e0bfd45e553-kube-api-access-r4jtw\" (UniqueName: \"kubernetes.io/projected/3d57de51-eda3-4e97-b477-4e0bfd45e553-kube-api-access-r4jtw\") pod \"3d57de51-eda3-4e97-b477-4e0bfd45e553\" (UID: \"3d57de51-eda3-4e97-b477-4e0bfd45e553\") "
Nov 01 22:50:56.672763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.672769    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d57de51-eda3-4e97-b477-4e0bfd45e553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d57de51-eda3-4e97-b477-4e0bfd45e553-plugins-config-volume\") pod \"3d57de51-eda3-4e97-b477-4e0bfd45e553\" (UID: \"3d57de51-eda3-4e97-b477-4e0bfd45e553\") "
Nov 01 22:50:56.673224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.673185    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d57de51-eda3-4e97-b477-4e0bfd45e553-plugins-config-volume" pod "3d57de51-eda3-4e97-b477-4e0bfd45e553" (UID: "3d57de51-eda3-4e97-b477-4e0bfd45e553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:50:56.673586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.673562    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d57de51-eda3-4e97-b477-4e0bfd45e553-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:56.675289 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.675264    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d57de51-eda3-4e97-b477-4e0bfd45e553-kube-api-access-r4jtw" pod "3d57de51-eda3-4e97-b477-4e0bfd45e553" (UID: "3d57de51-eda3-4e97-b477-4e0bfd45e553"). InnerVolumeSpecName "kube-api-access-r4jtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:50:56.700545 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.700512    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:50:56.775123 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.775096    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dce20b5-7772-4154-b4e6-70b379c1e439-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dce20b5-7772-4154-b4e6-70b379c1e439-plugins-config-volume\") pod \"8dce20b5-7772-4154-b4e6-70b379c1e439\" (UID: \"8dce20b5-7772-4154-b4e6-70b379c1e439\") "
Nov 01 22:50:56.775303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.775289    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dce20b5-7772-4154-b4e6-70b379c1e439-kube-api-access-nrk7j\" (UniqueName: \"kubernetes.io/projected/8dce20b5-7772-4154-b4e6-70b379c1e439-kube-api-access-nrk7j\") pod \"8dce20b5-7772-4154-b4e6-70b379c1e439\" (UID: \"8dce20b5-7772-4154-b4e6-70b379c1e439\") "
Nov 01 22:50:56.775459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.775439    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dce20b5-7772-4154-b4e6-70b379c1e439-plugins-config-volume" pod "8dce20b5-7772-4154-b4e6-70b379c1e439" (UID: "8dce20b5-7772-4154-b4e6-70b379c1e439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:50:56.775918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.775897    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4jtw\" (UniqueName: \"kubernetes.io/projected/3d57de51-eda3-4e97-b477-4e0bfd45e553-kube-api-access-r4jtw\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:56.775918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.775920    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dce20b5-7772-4154-b4e6-70b379c1e439-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:56.777370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.777351    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dce20b5-7772-4154-b4e6-70b379c1e439-kube-api-access-nrk7j" pod "8dce20b5-7772-4154-b4e6-70b379c1e439" (UID: "8dce20b5-7772-4154-b4e6-70b379c1e439"). InnerVolumeSpecName "kube-api-access-nrk7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:50:56.877830 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:56.877749    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrk7j\" (UniqueName: \"kubernetes.io/projected/8dce20b5-7772-4154-b4e6-70b379c1e439-kube-api-access-nrk7j\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:57.142969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.142888    4837 generic.go:356] "Generic (PLEG): container finished" podID="8dce20b5-7772-4154-b4e6-70b379c1e439" containerID="fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1" exitCode=0
Nov 01 22:50:57.142969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.142931    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"
Nov 01 22:50:57.143159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.142976    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b" event={"ID":"8dce20b5-7772-4154-b4e6-70b379c1e439","Type":"ContainerDied","Data":"fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1"}
Nov 01 22:50:57.143159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.143026    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b" event={"ID":"8dce20b5-7772-4154-b4e6-70b379c1e439","Type":"ContainerDied","Data":"81366c6a51506308335ea47d32875d149bed3106198f20c8cee6a400ae0711f6"}
Nov 01 22:50:57.143159 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.143071    4837 scope.go:122] "RemoveContainer" containerID="fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1"
Nov 01 22:50:57.144601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.144577    4837 generic.go:356] "Generic (PLEG): container finished" podID="3d57de51-eda3-4e97-b477-4e0bfd45e553" containerID="3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc" exitCode=0
Nov 01 22:50:57.144717 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.144611    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r" event={"ID":"3d57de51-eda3-4e97-b477-4e0bfd45e553","Type":"ContainerDied","Data":"3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc"}
Nov 01 22:50:57.144717 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.144670    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r" event={"ID":"3d57de51-eda3-4e97-b477-4e0bfd45e553","Type":"ContainerDied","Data":"81ea7b66cf4ef1f0665801a19caafe5e91e5d3938eceb88537021ce4f4a3a346"}
Nov 01 22:50:57.144717 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.144622    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"
Nov 01 22:50:57.154116 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.154086    4837 scope.go:122] "RemoveContainer" containerID="fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1"
Nov 01 22:50:57.154355 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:50:57.154337    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1\": container with ID starting with fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1 not found: ID does not exist" containerID="fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1"
Nov 01 22:50:57.154434 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.154362    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1"} err="failed to get container status \"fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1\": rpc error: code = NotFound desc = could not find container \"fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1\": container with ID starting with fbaf4324af703f4891f4acfd34df6783c12f7668e6bbd53d13ba9692946bacf1 not found: ID does not exist"
Nov 01 22:50:57.154434 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.154378    4837 scope.go:122] "RemoveContainer" containerID="3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc"
Nov 01 22:50:57.164368 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.164347    4837 scope.go:122] "RemoveContainer" containerID="3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc"
Nov 01 22:50:57.164622 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:50:57.164605    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc\": container with ID starting with 3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc not found: ID does not exist" containerID="3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc"
Nov 01 22:50:57.164675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.164628    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc"} err="failed to get container status \"3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc\": rpc error: code = NotFound desc = could not find container \"3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc\": container with ID starting with 3c34e60014cb62e7146d0a57ddb38822dfe3ed3d8f910a66a5b437fe673c5efc not found: ID does not exist"
Nov 01 22:50:57.299630 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.299602    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"]
Nov 01 22:50:57.303453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.303431    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68a7f708-5b66bd5465-8hx5r"]
Nov 01 22:50:57.404504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.404435    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"]
Nov 01 22:50:57.408360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:57.408339    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68a7f708-86d8bb5d7-mz26b"]
Nov 01 22:50:58.534848 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:58.534812    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d57de51-eda3-4e97-b477-4e0bfd45e553" path="/var/lib/kubelet/pods/3d57de51-eda3-4e97-b477-4e0bfd45e553/volumes"
Nov 01 22:50:58.535420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:50:58.535403    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dce20b5-7772-4154-b4e6-70b379c1e439" path="/var/lib/kubelet/pods/8dce20b5-7772-4154-b4e6-70b379c1e439/volumes"
Nov 01 22:51:06.587274 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.587243    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"]
Nov 01 22:51:06.587918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.587904    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3d57de51-eda3-4e97-b477-4e0bfd45e553" containerName="payload-processing"
Nov 01 22:51:06.587967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.587919    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d57de51-eda3-4e97-b477-4e0bfd45e553" containerName="payload-processing"
Nov 01 22:51:06.588002 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.587967    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8dce20b5-7772-4154-b4e6-70b379c1e439" containerName="payload-pre-processing"
Nov 01 22:51:06.588002 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.587972    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dce20b5-7772-4154-b4e6-70b379c1e439" containerName="payload-pre-processing"
Nov 01 22:51:06.588098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.588087    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d57de51-eda3-4e97-b477-4e0bfd45e553" containerName="payload-processing"
Nov 01 22:51:06.588133 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.588109    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dce20b5-7772-4154-b4e6-70b379c1e439" containerName="payload-pre-processing"
Nov 01 22:51:06.593387 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.593370    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.673076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.673032    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"]
Nov 01 22:51:06.690972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.690949    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-945efd02\"/\"openshift-service-ca.crt\""
Nov 01 22:51:06.691152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.691055    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-945efd02\"/\"e2e-lifecycle-model-945efd02-kserve-self-signed-certs\""
Nov 01 22:51:06.691152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.691074    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-945efd02\"/\"kube-root-ca.crt\""
Nov 01 22:51:06.691284 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.691153    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-945efd02\"/\"default-dockercfg-7d8sb\""
Nov 01 22:51:06.780191 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.780148    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhvv\" (UniqueName: \"kubernetes.io/projected/e8ebd46d-560f-40e6-b31a-2946fc22c11d-kube-api-access-7fhvv\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.780568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.780548    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-dshm\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.820965 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.820925    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-home\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.821707 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.821682    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tls-certs\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.821925 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.821906    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-model-cache\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.822209 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.822182    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tmp-dir\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.937263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.937216    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-dshm\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.937622 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.937599    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-home\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.937958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.937936    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tls-certs\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.938164 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.938143    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-model-cache\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.938334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.938319    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tmp-dir\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.950188 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.950156    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhvv\" (UniqueName: \"kubernetes.io/projected/e8ebd46d-560f-40e6-b31a-2946fc22c11d-kube-api-access-7fhvv\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.950573 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.942634    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-home\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.997066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.942823    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tmp-dir\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.997066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.942426    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-model-cache\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.997066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.967452    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-dshm\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:06.999685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:06.998033    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tls-certs\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:07.028061 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:07.028008    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhvv\" (UniqueName: \"kubernetes.io/projected/e8ebd46d-560f-40e6-b31a-2946fc22c11d-kube-api-access-7fhvv\") pod \"e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") " pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:07.211121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:07.207415    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:07.785709 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:51:07.785684    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8ebd46d_560f_40e6_b31a_2946fc22c11d.slice/crio-2198dcf82dc94f5b3da7515cb14e5d876df243fbfc033491d536cb7a256f7758 WatchSource:0}: Error finding container 2198dcf82dc94f5b3da7515cb14e5d876df243fbfc033491d536cb7a256f7758: Status 404 returned error can't find the container with id 2198dcf82dc94f5b3da7515cb14e5d876df243fbfc033491d536cb7a256f7758
Nov 01 22:51:07.786058 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:07.785856    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"]
Nov 01 22:51:08.333125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:08.333093    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" event={"ID":"e8ebd46d-560f-40e6-b31a-2946fc22c11d","Type":"ContainerStarted","Data":"83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd"}
Nov 01 22:51:08.333297 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:08.333135    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" event={"ID":"e8ebd46d-560f-40e6-b31a-2946fc22c11d","Type":"ContainerStarted","Data":"2198dcf82dc94f5b3da7515cb14e5d876df243fbfc033491d536cb7a256f7758"}
Nov 01 22:51:08.443230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:08.443183    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" podStartSLOduration=2.443168162 podStartE2EDuration="2.443168162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:08.440915341 +0000 UTC m=+1650.030114402" watchObservedRunningTime="2026-11-01 22:51:08.443168162 +0000 UTC m=+1650.032367261"
Nov 01 22:51:09.208708 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:09.208670    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:09.671230 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:09.671192    4837 patch_prober.go:29] interesting pod/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:51:09.671397 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:09.671273    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:51:12.369255 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:12.369163    4837 generic.go:356] "Generic (PLEG): container finished" podID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerID="055aaae705f7618bdd13077ab1766012f0de79cf7f423f65bfc26c20d02d2677" exitCode=0
Nov 01 22:51:12.369255 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:12.369237    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" event={"ID":"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7","Type":"ContainerDied","Data":"055aaae705f7618bdd13077ab1766012f0de79cf7f423f65bfc26c20d02d2677"}
Nov 01 22:51:12.369781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:12.369654    4837 scope.go:122] "RemoveContainer" containerID="055aaae705f7618bdd13077ab1766012f0de79cf7f423f65bfc26c20d02d2677"
Nov 01 22:51:12.371474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:12.371447    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:12.371543 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:12.371518    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:12.371587 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:12.371551    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:13.378810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:13.378778    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" event={"ID":"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7","Type":"ContainerStarted","Data":"570655f006461514baf5c3eba7644022df3430c19b26d3ad1d95f256b6d0de39"}
Nov 01 22:51:13.408872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:13.408846    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:51:13.409024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:13.408994    4837 patch_prober.go:29] interesting pod/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 01 22:51:13.409084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:13.409033    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 01 22:51:14.408791 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:14.408755    4837 patch_prober.go:29] interesting pod/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 01 22:51:14.409225 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:14.408810    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 01 22:51:15.058448 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:15.058397    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"]
Nov 01 22:51:15.101179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:15.101140    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4784835-2m24n"]
Nov 01 22:51:15.408866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:15.408785    4837 patch_prober.go:29] interesting pod/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 01 22:51:15.408866 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:15.408831    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 01 22:51:16.409570 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:16.409537    4837 patch_prober.go:29] interesting pod/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 01 22:51:16.409955 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:16.409594    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 01 22:51:16.529363 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:16.528482    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6" path="/var/lib/kubelet/pods/9eb22bef-6b39-4bad-9bbf-a2a54d2e5ee6/volumes"
Nov 01 22:51:17.208954 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:17.208921    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:17.221770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:17.221747    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:17.409250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:17.409218    4837 patch_prober.go:29] interesting pod/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 01 22:51:17.409447 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:17.409285    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 01 22:51:17.425004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:17.424977    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:51:18.409497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:18.409429    4837 patch_prober.go:29] interesting pod/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 01 22:51:18.409736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:18.409710    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 01 22:51:18.459748 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:18.459716    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"]
Nov 01 22:51:18.460153 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:18.459947    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76" podUID="f475bb62-cd9f-43e0-b045-f50944e6d69e" containerName="maas-api" containerID="cri-o://4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c" gracePeriod=30
Nov 01 22:51:18.978097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:18.970071    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:51:19.099089 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:19.099038    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:51:19.195115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.194377    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f475bb62-cd9f-43e0-b045-f50944e6d69e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f475bb62-cd9f-43e0-b045-f50944e6d69e-openshift-service-ca\") pod \"f475bb62-cd9f-43e0-b045-f50944e6d69e\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") "
Nov 01 22:51:19.195115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.194455    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f475bb62-cd9f-43e0-b045-f50944e6d69e-kube-api-access-9tpvs\" (UniqueName: \"kubernetes.io/projected/f475bb62-cd9f-43e0-b045-f50944e6d69e-kube-api-access-9tpvs\") pod \"f475bb62-cd9f-43e0-b045-f50944e6d69e\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") "
Nov 01 22:51:19.195115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.194542    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-maas-api-tls\") pod \"f475bb62-cd9f-43e0-b045-f50944e6d69e\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") "
Nov 01 22:51:19.195115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.194782    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-metrics-certs\") pod \"f475bb62-cd9f-43e0-b045-f50944e6d69e\" (UID: \"f475bb62-cd9f-43e0-b045-f50944e6d69e\") "
Nov 01 22:51:19.222067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.207429    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f475bb62-cd9f-43e0-b045-f50944e6d69e-openshift-service-ca" pod "f475bb62-cd9f-43e0-b045-f50944e6d69e" (UID: "f475bb62-cd9f-43e0-b045-f50944e6d69e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:19.222067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.217648    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-metrics-certs" pod "f475bb62-cd9f-43e0-b045-f50944e6d69e" (UID: "f475bb62-cd9f-43e0-b045-f50944e6d69e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:51:19.263857 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.263812    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f475bb62-cd9f-43e0-b045-f50944e6d69e-kube-api-access-9tpvs" pod "f475bb62-cd9f-43e0-b045-f50944e6d69e" (UID: "f475bb62-cd9f-43e0-b045-f50944e6d69e"). InnerVolumeSpecName "kube-api-access-9tpvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:19.275264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.275185    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-maas-api-tls" pod "f475bb62-cd9f-43e0-b045-f50944e6d69e" (UID: "f475bb62-cd9f-43e0-b045-f50944e6d69e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:51:19.297470 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.297444    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f475bb62-cd9f-43e0-b045-f50944e6d69e-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:19.297470 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.297469    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tpvs\" (UniqueName: \"kubernetes.io/projected/f475bb62-cd9f-43e0-b045-f50944e6d69e-kube-api-access-9tpvs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:19.297710 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.297479    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:19.297710 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.297488    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f475bb62-cd9f-43e0-b045-f50944e6d69e-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:19.412667 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.412635    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:51:19.413003 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.412982    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:51:19.413758 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.413732    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:51:19.429282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.429255    4837 generic.go:356] "Generic (PLEG): container finished" podID="f475bb62-cd9f-43e0-b045-f50944e6d69e" containerID="4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c" exitCode=0
Nov 01 22:51:19.429386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.429299    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"
Nov 01 22:51:19.429386 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.429338    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76" event={"ID":"f475bb62-cd9f-43e0-b045-f50944e6d69e","Type":"ContainerDied","Data":"4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c"}
Nov 01 22:51:19.429475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.429390    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76" event={"ID":"f475bb62-cd9f-43e0-b045-f50944e6d69e","Type":"ContainerDied","Data":"2b3b35d47ade3abad2510bc0026f0cf27bc68b9298cbf65b39e0f97221148a0e"}
Nov 01 22:51:19.429475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.429417    4837 scope.go:122] "RemoveContainer" containerID="4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c"
Nov 01 22:51:19.441003 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.440978    4837 scope.go:122] "RemoveContainer" containerID="4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c"
Nov 01 22:51:19.441309 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:19.441288    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c\": container with ID starting with 4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c not found: ID does not exist" containerID="4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c"
Nov 01 22:51:19.441382 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.441316    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c"} err="failed to get container status \"4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c\": rpc error: code = NotFound desc = could not find container \"4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c\": container with ID starting with 4572e3030a5224a02083f2182852d5d712463a6845c683dc952dfdebdba18b4c not found: ID does not exist"
Nov 01 22:51:19.752532 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.752460    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"]
Nov 01 22:51:19.757774 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:19.757749    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d04c71aa-64dc948589-r4m76"]
Nov 01 22:51:20.550865 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:20.548908    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f475bb62-cd9f-43e0-b045-f50944e6d69e" path="/var/lib/kubelet/pods/f475bb62-cd9f-43e0-b045-f50944e6d69e/volumes"
Nov 01 22:51:20.809166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:20.809085    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"]
Nov 01 22:51:20.809623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:20.809348    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy" containerID="cri-o://b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93" gracePeriod=30
Nov 01 22:51:22.086150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.086114    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"]
Nov 01 22:51:22.086858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.086831    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f475bb62-cd9f-43e0-b045-f50944e6d69e" containerName="maas-api"
Nov 01 22:51:22.086858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.086852    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f475bb62-cd9f-43e0-b045-f50944e6d69e" containerName="maas-api"
Nov 01 22:51:22.087058 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.087014    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f475bb62-cd9f-43e0-b045-f50944e6d69e" containerName="maas-api"
Nov 01 22:51:22.096304 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.096282    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.161703 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.161669    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-646881-openshift-default-dockercfg-pv67q\""
Nov 01 22:51:22.319069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.270366    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-certs\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.340139    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-data\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.340190    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-podinfo\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.340250    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-socket\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.340451    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-envoy\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.340562    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-credential-socket\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.340615    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqb8l\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-kube-api-access-nqb8l\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.340808    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-token\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.340865    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istiod-ca-cert\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.361118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.306417    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"]
Nov 01 22:51:22.480829 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.480788    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-envoy\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.492070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.481157    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-credential-socket\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.492407 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.492375    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqb8l\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-kube-api-access-nqb8l\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.492914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.492886    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-token\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.493163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.493142    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istiod-ca-cert\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.493360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.493342    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-certs\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.493492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.493474    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-data\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.493613 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.493598    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-podinfo\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.493759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.493744    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-socket\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.494799 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.494774    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-credential-socket\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.501197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.498303    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istiod-ca-cert\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.520777 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.502070    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-certs\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.597071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.532708    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-data\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.726314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.558238    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-socket\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.737420 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.737381    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqb8l\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-kube-api-access-nqb8l\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.741659 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.740861    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-podinfo\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.753066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.751646    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-envoy\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.758792 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.758760    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-token\") pod \"e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") " pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:22.844152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:22.840692    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:23.267830 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:51:23.267800    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2baff4f5_c9e3_4860_9a29_2b36a1e09547.slice/crio-1c7b38654260c23a54f8e01282d5fbba2e40f63b6f01dbb96970d14218f8aad6 WatchSource:0}: Error finding container 1c7b38654260c23a54f8e01282d5fbba2e40f63b6f01dbb96970d14218f8aad6: Status 404 returned error can't find the container with id 1c7b38654260c23a54f8e01282d5fbba2e40f63b6f01dbb96970d14218f8aad6
Nov 01 22:51:23.268224 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:23.268130    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"]
Nov 01 22:51:23.270657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:23.270627    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:23.270749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:23.270691    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:23.270749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:23.270719    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:23.576391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:23.576300    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" event={"ID":"2baff4f5-c9e3-4860-9a29-2b36a1e09547","Type":"ContainerStarted","Data":"f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363"}
Nov 01 22:51:23.576391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:23.576358    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" event={"ID":"2baff4f5-c9e3-4860-9a29-2b36a1e09547","Type":"ContainerStarted","Data":"1c7b38654260c23a54f8e01282d5fbba2e40f63b6f01dbb96970d14218f8aad6"}
Nov 01 22:51:23.693981 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:23.693925    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" podStartSLOduration=1.693911898 podStartE2EDuration="1.693911898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:23.68892732 +0000 UTC m=+1665.278126381" watchObservedRunningTime="2026-11-01 22:51:23.693911898 +0000 UTC m=+1665.283110958"
Nov 01 22:51:23.853113 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:23.845899    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:24.848595 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:24.848565    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:25.619280 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:25.619248    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:25.620196 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:25.620165    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:51:26.397848 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.397813    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:51:26.551177 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551098    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-data\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.551344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551222    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-envoy\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.551344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551252    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-socket\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.551344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551277    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-istio-token\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-istio-token\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.551510 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551337    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-data" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:26.551510 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551343    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7646a63-328d-4629-bd52-0f0f50486d82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7646a63-328d-4629-bd52-0f0f50486d82-istiod-ca-cert\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.551631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551506    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-credential-socket\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.551631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551583    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-socket" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:26.551631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551611    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-kube-api-access-xx492\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-kube-api-access-xx492\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.552687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551686    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7646a63-328d-4629-bd52-0f0f50486d82-istiod-ca-cert" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:26.552687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551708    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-certs\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.552687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551844    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7646a63-328d-4629-bd52-0f0f50486d82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7646a63-328d-4629-bd52-0f0f50486d82-istio-podinfo\") pod \"e7646a63-328d-4629-bd52-0f0f50486d82\" (UID: \"e7646a63-328d-4629-bd52-0f0f50486d82\") "
Nov 01 22:51:26.552687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.551840    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-credential-socket" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:26.552687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.552088    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-certs" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:26.553000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.552745    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.553000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.552769    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.553000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.552783    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.553000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.552797    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7646a63-328d-4629-bd52-0f0f50486d82-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.553000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.552811    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.553855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.553828    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-envoy" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:26.553963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.553920    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-istio-token" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:26.553963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.553939    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-kube-api-access-xx492" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "kube-api-access-xx492". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:26.554248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.554231    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7646a63-328d-4629-bd52-0f0f50486d82-istio-podinfo" pod "e7646a63-328d-4629-bd52-0f0f50486d82" (UID: "e7646a63-328d-4629-bd52-0f0f50486d82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:26.628894 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.628862    4837 generic.go:356] "Generic (PLEG): container finished" podID="e7646a63-328d-4629-bd52-0f0f50486d82" containerID="b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93" exitCode=0
Nov 01 22:51:26.629092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.628914    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"
Nov 01 22:51:26.629092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.628945    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" event={"ID":"e7646a63-328d-4629-bd52-0f0f50486d82","Type":"ContainerDied","Data":"b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93"}
Nov 01 22:51:26.629092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.629005    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq" event={"ID":"e7646a63-328d-4629-bd52-0f0f50486d82","Type":"ContainerDied","Data":"16528e40478aee03593079bb90086a581875418cc2541012e678e62fc2bf7d22"}
Nov 01 22:51:26.629092 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.629035    4837 scope.go:122] "RemoveContainer" containerID="b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93"
Nov 01 22:51:26.641495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.641478    4837 scope.go:122] "RemoveContainer" containerID="d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8"
Nov 01 22:51:26.655759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.655729    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.655759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.655764    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx492\" (UniqueName: \"kubernetes.io/projected/e7646a63-328d-4629-bd52-0f0f50486d82-kube-api-access-xx492\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.655998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.655781    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7646a63-328d-4629-bd52-0f0f50486d82-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.655998 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.655795    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7646a63-328d-4629-bd52-0f0f50486d82-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.656458 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.656442    4837 scope.go:122] "RemoveContainer" containerID="b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93"
Nov 01 22:51:26.656726 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:26.656708    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93\": container with ID starting with b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93 not found: ID does not exist" containerID="b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93"
Nov 01 22:51:26.656770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.656734    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93"} err="failed to get container status \"b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93\": rpc error: code = NotFound desc = could not find container \"b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93\": container with ID starting with b326d67e2098963b6dcb84383c30f1110685920ac91879625506caa40ccf7c93 not found: ID does not exist"
Nov 01 22:51:26.656770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.656750    4837 scope.go:122] "RemoveContainer" containerID="d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8"
Nov 01 22:51:26.657022 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:26.657006    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8\": container with ID starting with d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8 not found: ID does not exist" containerID="d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8"
Nov 01 22:51:26.657109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.657026    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8"} err="failed to get container status \"d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8\": rpc error: code = NotFound desc = could not find container \"d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8\": container with ID starting with d4b147d460110701fd0c64a7f9c24a4dadf1da38dd666d223fdb0facce41e5f8 not found: ID does not exist"
Nov 01 22:51:26.787004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.786970    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"]
Nov 01 22:51:26.790898 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:26.790871    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d04c71aa-gw-openshift-default-5fd65566fc-82csq"]
Nov 01 22:51:27.084931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:27.084439    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"]
Nov 01 22:51:27.084931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:27.084690    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy" containerID="cri-o://f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9" gracePeriod=30
Nov 01 22:51:27.876470 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:27.876425    4837 patch_prober.go:29] interesting pod/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:27.876871 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:27.876505    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:28.439847 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.439816    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"]
Nov 01 22:51:28.440570 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.440555    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy"
Nov 01 22:51:28.440570 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.440570    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy"
Nov 01 22:51:28.440666 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.440591    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy"
Nov 01 22:51:28.440666 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.440601    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy"
Nov 01 22:51:28.440753 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.440742    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy"
Nov 01 22:51:28.440786 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.440757    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" containerName="istio-proxy"
Nov 01 22:51:28.444343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.444322    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.498450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.498418    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8bac2f-openshift-default-dockercfg-dvrdf\""
Nov 01 22:51:28.580498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.580452    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6rw\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-kube-api-access-gt6rw\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.747767 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.747684    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d77d32-49a4-405f-af93-db6b4fc0e303-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.748683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.748656    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.748936 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.748918    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.749269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.749246    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.749552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.749533    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.749716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.749701    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.749943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.749928    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.750148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.750130    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.782018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.780490    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7646a63-328d-4629-bd52-0f0f50486d82" path="/var/lib/kubelet/pods/e7646a63-328d-4629-bd52-0f0f50486d82/volumes"
Nov 01 22:51:28.783562 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.782640    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"]
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853061    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853170    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853209    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853308    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853368    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853539    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6rw\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-kube-api-access-gt6rw\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853611    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d77d32-49a4-405f-af93-db6b4fc0e303-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853665    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.853722    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.858351    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.858539    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.865152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.858692    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.871722 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.871579    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d77d32-49a4-405f-af93-db6b4fc0e303-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.874034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.872238    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.880005 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.879982    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.952154 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.939561    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6rw\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-kube-api-access-gt6rw\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.952154 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.942658    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:28.958834 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:28.958798    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:29.063163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:29.058804    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:29.637128 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:51:29.637100    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d77d32_49a4_405f_af93_db6b4fc0e303.slice/crio-b604b334671d789eb7135ad4bcdd5b2c127d1b035c6ade7b98089429da85aab5 WatchSource:0}: Error finding container b604b334671d789eb7135ad4bcdd5b2c127d1b035c6ade7b98089429da85aab5: Status 404 returned error can't find the container with id b604b334671d789eb7135ad4bcdd5b2c127d1b035c6ade7b98089429da85aab5
Nov 01 22:51:29.638157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:29.638130    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"]
Nov 01 22:51:29.639992 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:29.639961    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:29.640083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:29.640024    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:29.640142 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:29.640081    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:29.755916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:29.755884    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" event={"ID":"70d77d32-49a4-405f-af93-db6b4fc0e303","Type":"ContainerStarted","Data":"3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6"}
Nov 01 22:51:29.756117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:29.755940    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" event={"ID":"70d77d32-49a4-405f-af93-db6b4fc0e303","Type":"ContainerStarted","Data":"b604b334671d789eb7135ad4bcdd5b2c127d1b035c6ade7b98089429da85aab5"}
Nov 01 22:51:29.876381 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:29.876318    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podStartSLOduration=1.876298045 podStartE2EDuration="1.876298045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:29.876149285 +0000 UTC m=+1671.465348346" watchObservedRunningTime="2026-11-01 22:51:29.876298045 +0000 UTC m=+1671.465497107"
Nov 01 22:51:30.080222 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:30.074750    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:30.080222 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:30.074883    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:51:30.080222 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:30.074950    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:51:31.191065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:31.188361    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:31.965645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:31.962551    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:31.965645 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:31.963707    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:51:32.500456 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.500427    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:51:32.678548 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678522    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-envoy\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.678726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678564    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-socket\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.678726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678594    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-token\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-token\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.678726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678625    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-credential-socket\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.678726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678712    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-data\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.678941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678740    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istiod-ca-cert\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.678941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678792    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-podinfo\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.678941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678842    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-socket" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:32.678941 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.678902    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-certs\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.679210 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.679036    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-kube-api-access-bq8dl\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-kube-api-access-bq8dl\") pod \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\" (UID: \"7119d22b-44f1-4c43-8a7b-c1dc954edaab\") "
Nov 01 22:51:32.679210 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.679100    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-data" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:32.679210 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.679103    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-credential-socket" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:32.679368 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.679262    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-certs" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:32.679520 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.679480    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istiod-ca-cert" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:32.680169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.680142    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.680282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.680170    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.680282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.680185    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.680282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.680201    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.680282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.680215    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.681120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.681094    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-envoy" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:32.681214 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.681164    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-token" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:32.681214 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.681191    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-podinfo" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:32.681639 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.681617    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-kube-api-access-bq8dl" pod "7119d22b-44f1-4c43-8a7b-c1dc954edaab" (UID: "7119d22b-44f1-4c43-8a7b-c1dc954edaab"). InnerVolumeSpecName "kube-api-access-bq8dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:32.781530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.781492    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.781530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.781523    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.781530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.781533    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7119d22b-44f1-4c43-8a7b-c1dc954edaab-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.781752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.781542    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq8dl\" (UniqueName: \"kubernetes.io/projected/7119d22b-44f1-4c43-8a7b-c1dc954edaab-kube-api-access-bq8dl\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:32.919715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.919685    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"]
Nov 01 22:51:32.920158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.920126    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy" containerID="cri-o://c385554dff7fdcb7278617177272e1ca4da51ce80898e3e371d340fadd4f4ec3" gracePeriod=30
Nov 01 22:51:32.956233 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.946912    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"]
Nov 01 22:51:32.956233 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.947236    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy" containerID="cri-o://42290030f6164fb3169243b25edefd856732539951637bf718ff94aa5a740378" gracePeriod=30
Nov 01 22:51:33.000080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:32.995788    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"]
Nov 01 22:51:33.050121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.047837    4837 generic.go:356] "Generic (PLEG): container finished" podID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerID="f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9" exitCode=0
Nov 01 22:51:33.050121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.049232    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"
Nov 01 22:51:33.055798 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.055763    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" event={"ID":"7119d22b-44f1-4c43-8a7b-c1dc954edaab","Type":"ContainerDied","Data":"f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9"}
Nov 01 22:51:33.055953 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.055817    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825" event={"ID":"7119d22b-44f1-4c43-8a7b-c1dc954edaab","Type":"ContainerDied","Data":"525bbe5cd1621be2ceeed473b03cbfc7ed19b3a7646c28ba0b07a2cb0b7ab64e"}
Nov 01 22:51:33.055953 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.055848    4837 scope.go:122] "RemoveContainer" containerID="f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9"
Nov 01 22:51:33.056202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.056170    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy" containerID="cri-o://530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6" gracePeriod=30
Nov 01 22:51:33.058673 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.058649    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"]
Nov 01 22:51:33.058970 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.058904    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" containerID="cri-o://2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f" gracePeriod=30
Nov 01 22:51:33.076134 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.074478    4837 scope.go:122] "RemoveContainer" containerID="8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62"
Nov 01 22:51:33.089563 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.089539    4837 scope.go:122] "RemoveContainer" containerID="f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9"
Nov 01 22:51:33.089902 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:33.089874    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9\": container with ID starting with f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9 not found: ID does not exist" containerID="f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9"
Nov 01 22:51:33.089990 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.089917    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9"} err="failed to get container status \"f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9\": rpc error: code = NotFound desc = could not find container \"f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9\": container with ID starting with f6b88bb22aa949f237c8c8cd3a0fd64ad4e0cf3c8ca4d8638195e04f99d9c1f9 not found: ID does not exist"
Nov 01 22:51:33.089990 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.089945    4837 scope.go:122] "RemoveContainer" containerID="8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62"
Nov 01 22:51:33.091334 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:33.091302    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62\": container with ID starting with 8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62 not found: ID does not exist" containerID="8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62"
Nov 01 22:51:33.091440 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.091343    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62"} err="failed to get container status \"8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62\": rpc error: code = NotFound desc = could not find container \"8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62\": container with ID starting with 8aea27ee9d5a570fef9d94ed24dfa6af1f8a0f591dd9c453d1a2d8d89397df62 not found: ID does not exist"
Nov 01 22:51:33.271201 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.271080    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"]
Nov 01 22:51:33.282461 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:33.282424    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-68a7f708-openshift-default-5fcdddc89-mh825"]
Nov 01 22:51:34.531714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:34.531678    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" path="/var/lib/kubelet/pods/7119d22b-44f1-4c43-8a7b-c1dc954edaab/volumes"
Nov 01 22:51:35.307071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.304341    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"]
Nov 01 22:51:35.307071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.305774    4837 cpu_manager.go:414] "containerMap: removing container" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy"
Nov 01 22:51:35.307071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.305806    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy"
Nov 01 22:51:35.307071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.305871    4837 cpu_manager.go:414] "containerMap: removing container" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy"
Nov 01 22:51:35.307071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.305882    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy"
Nov 01 22:51:35.315345 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.315314    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy"
Nov 01 22:51:35.336130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.331356    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.369665 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.369407    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"]
Nov 01 22:51:35.414069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.406760    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-2b0866-openshift-default-dockercfg-z9htn\""
Nov 01 22:51:35.496077 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.472320    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-token\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.545108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.542141    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-socket\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.545108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.542225    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-certs\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.545108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.542354    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-envoy\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.802208 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.802130    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s58t\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-kube-api-access-4s58t\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.802526 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.802499    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-credential-socket\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.804326 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.804302    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-podinfo\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.804536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.804505    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istiod-ca-cert\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.805288 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.805260    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-data\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.935451 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.935412    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-podinfo\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.969316    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istiod-ca-cert\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.969395    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-data\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.969603    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-token\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.969669    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-socket\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.969722    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-certs\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.969859    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-envoy\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.969984    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s58t\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-kube-api-access-4s58t\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.970076    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-credential-socket\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.970349    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istiod-ca-cert\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.971194    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.971239    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.971880    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-credential-socket\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.972098    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-socket\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.978121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.972568    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-certs\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:35.982576 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:35.982531    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-data\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:36.019078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.009952    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-envoy\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:36.032077 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.030650    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s58t\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-kube-api-access-4s58t\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:36.032077 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.031492    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-podinfo\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:36.062124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.035211    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-token\") pod \"e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") " pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:36.139559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.139522    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:36.497100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.497064    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:36.497292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.497130    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:36.595700 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:51:36.595666    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b56f6a2_6dad_405e_9d88_c30f0f9cee77.slice/crio-cf5ad827866e035d4b5f02ed5fe80d5ad65340c7180b4812de5fa96bac78a2fa WatchSource:0}: Error finding container cf5ad827866e035d4b5f02ed5fe80d5ad65340c7180b4812de5fa96bac78a2fa: Status 404 returned error can't find the container with id cf5ad827866e035d4b5f02ed5fe80d5ad65340c7180b4812de5fa96bac78a2fa
Nov 01 22:51:36.598703 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.598673    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:36.598787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.598743    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:36.598787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.598782    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:51:36.601841 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:36.601826    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"]
Nov 01 22:51:37.145846 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:37.145811    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" event={"ID":"6b56f6a2-6dad-405e-9d88-c30f0f9cee77","Type":"ContainerStarted","Data":"56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c"}
Nov 01 22:51:37.146101 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:37.145875    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" event={"ID":"6b56f6a2-6dad-405e-9d88-c30f0f9cee77","Type":"ContainerStarted","Data":"cf5ad827866e035d4b5f02ed5fe80d5ad65340c7180b4812de5fa96bac78a2fa"}
Nov 01 22:51:37.264082 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:37.262736    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" podStartSLOduration=2.26271644 podStartE2EDuration="2.26271644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:37.2584741 +0000 UTC m=+1678.847673161" watchObservedRunningTime="2026-11-01 22:51:37.26271644 +0000 UTC m=+1678.851915502"
Nov 01 22:51:37.872085 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:37.872012    4837 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:37.872471 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:37.872099    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:38.139991 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.139913    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:38.158147 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.149354    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:38.160575 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.160537    4837 generic.go:356] "Generic (PLEG): container finished" podID="28eac005-a391-4ba5-b018-c870d18a1033" containerID="42290030f6164fb3169243b25edefd856732539951637bf718ff94aa5a740378" exitCode=0
Nov 01 22:51:38.164423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.164390    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" event={"ID":"28eac005-a391-4ba5-b018-c870d18a1033","Type":"ContainerDied","Data":"42290030f6164fb3169243b25edefd856732539951637bf718ff94aa5a740378"}
Nov 01 22:51:38.169679 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.169644    4837 generic.go:356] "Generic (PLEG): container finished" podID="468199b3-71a0-4e45-927f-3dbe77075f56" containerID="c385554dff7fdcb7278617177272e1ca4da51ce80898e3e371d340fadd4f4ec3" exitCode=0
Nov 01 22:51:38.170137 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.170114    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" event={"ID":"468199b3-71a0-4e45-927f-3dbe77075f56","Type":"ContainerDied","Data":"c385554dff7fdcb7278617177272e1ca4da51ce80898e3e371d340fadd4f4ec3"}
Nov 01 22:51:38.171150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.171114    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:38.172080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.172037    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:51:38.302139 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.302032    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:51:38.335357 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.335331    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.470616    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468199b3-71a0-4e45-927f-3dbe77075f56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468199b3-71a0-4e45-927f-3dbe77075f56-istiod-ca-cert\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.470817    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-certs\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.470868    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-istio-token\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-istio-token\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.470905    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-socket\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.470933    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28eac005-a391-4ba5-b018-c870d18a1033-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28eac005-a391-4ba5-b018-c870d18a1033-istiod-ca-cert\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.471023    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/468199b3-71a0-4e45-927f-3dbe77075f56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468199b3-71a0-4e45-927f-3dbe77075f56-istio-podinfo\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.472521    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-certs" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.472869    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468199b3-71a0-4e45-927f-3dbe77075f56-istiod-ca-cert" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.473140    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-socket" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.474988    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28eac005-a391-4ba5-b018-c870d18a1033-istiod-ca-cert" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.477805    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468199b3-71a0-4e45-927f-3dbe77075f56-istio-podinfo" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.480350    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-istio-token" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497130    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-credential-socket\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497194    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-data\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497253    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-istio-token\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-istio-token\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497299    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-certs\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497338    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-envoy\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497367    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28eac005-a391-4ba5-b018-c870d18a1033-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28eac005-a391-4ba5-b018-c870d18a1033-istio-podinfo\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497452    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-credential-socket\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497478    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-envoy\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497506    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-kube-api-access-xvf9x\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-kube-api-access-xvf9x\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497537    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-data\") pod \"28eac005-a391-4ba5-b018-c870d18a1033\" (UID: \"28eac005-a391-4ba5-b018-c870d18a1033\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497597    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-kube-api-access-7lbtd\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-kube-api-access-7lbtd\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.497763 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.497700    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-socket\") pod \"468199b3-71a0-4e45-927f-3dbe77075f56\" (UID: \"468199b3-71a0-4e45-927f-3dbe77075f56\") "
Nov 01 22:51:38.499088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.498710    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.500574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.499522    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-data" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.500574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.500114    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-certs" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.500574 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.500340    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-data" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.501055 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.501019    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-credential-socket" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.501361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.501342    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-credential-socket" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.501705 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.501688    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.501823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.501811    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.501908 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.501897    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28eac005-a391-4ba5-b018-c870d18a1033-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.501991 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.501980    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468199b3-71a0-4e45-927f-3dbe77075f56-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.502098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.502084    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468199b3-71a0-4e45-927f-3dbe77075f56-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.502390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.502372    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-socket" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.521206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.520831    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-kube-api-access-xvf9x" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "kube-api-access-xvf9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:38.579344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.579313    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-envoy" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.583759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.580973    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-envoy" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.583759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.583472    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28eac005-a391-4ba5-b018-c870d18a1033-istio-podinfo" pod "28eac005-a391-4ba5-b018-c870d18a1033" (UID: "28eac005-a391-4ba5-b018-c870d18a1033"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:38.586702 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.586261    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-kube-api-access-7lbtd" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "kube-api-access-7lbtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:38.587295 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.586836    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-istio-token" pod "468199b3-71a0-4e45-927f-3dbe77075f56" (UID: "468199b3-71a0-4e45-927f-3dbe77075f56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612393    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612417    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28eac005-a391-4ba5-b018-c870d18a1033-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612428    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612437    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612446    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvf9x\" (UniqueName: \"kubernetes.io/projected/28eac005-a391-4ba5-b018-c870d18a1033-kube-api-access-xvf9x\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612456    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612464    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lbtd\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-kube-api-access-7lbtd\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612473    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612481    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612489    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468199b3-71a0-4e45-927f-3dbe77075f56-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612498    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468199b3-71a0-4e45-927f-3dbe77075f56-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.612506    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28eac005-a391-4ba5-b018-c870d18a1033-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.621827 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.620667    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:51:38.671342 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.671315    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:51:38.713959 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.713927    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-token\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-token\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714089    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-podinfo\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.714163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714128    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-data\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714164    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-podinfo\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714223    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-envoy\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.714293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714273    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-certs\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.714441 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714353    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-envoy\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714441 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714402    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-kube-api-access-7m2m7\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-kube-api-access-7m2m7\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714544 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714479    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-kube-api-access-xldpz\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-kube-api-access-xldpz\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.714544 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714527    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-token\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-token\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.714644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714563    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830f4875-c6ea-44d1-8d67-c38918d477a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830f4875-c6ea-44d1-8d67-c38918d477a4-istiod-ca-cert\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.714644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714587    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-certs\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714610    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-socket\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714789 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714663    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-data" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.714789 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714692    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-credential-socket\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.714789 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714740    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-data\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.714789 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714761    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-credential-socket\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714975 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714792    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istiod-ca-cert\") pod \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\" (UID: \"e2548cff-7724-4e35-a7a1-c97c40f7dc5f\") "
Nov 01 22:51:38.714975 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.714812    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-socket\") pod \"830f4875-c6ea-44d1-8d67-c38918d477a4\" (UID: \"830f4875-c6ea-44d1-8d67-c38918d477a4\") "
Nov 01 22:51:38.715636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.715616    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.715974    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-credential-socket" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.716310    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830f4875-c6ea-44d1-8d67-c38918d477a4-istiod-ca-cert" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.716535    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-certs" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.716731    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-socket" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.716889    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-credential-socket" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.717069    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-certs" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.717131    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-data" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.717304    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-token" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:38.717536 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.717374    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istiod-ca-cert" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:38.720588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.717565    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-token" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:38.720588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.717608    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-socket" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.720588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.718580    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-kube-api-access-7m2m7" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "kube-api-access-7m2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:38.720588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.718629    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-envoy" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.720588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.718895    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-podinfo" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:38.720588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.719588    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-podinfo" pod "e2548cff-7724-4e35-a7a1-c97c40f7dc5f" (UID: "e2548cff-7724-4e35-a7a1-c97c40f7dc5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:38.720588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.719917    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-envoy" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:38.720959 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.720638    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-kube-api-access-xldpz" pod "830f4875-c6ea-44d1-8d67-c38918d477a4" (UID: "830f4875-c6ea-44d1-8d67-c38918d477a4"). InnerVolumeSpecName "kube-api-access-xldpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831848    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831883    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831897    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831909    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831922    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831933    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831947    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831960    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831973    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.831987    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.832000    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m2m7\" (UniqueName: \"kubernetes.io/projected/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-kube-api-access-7m2m7\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.832014    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xldpz\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-kube-api-access-xldpz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.832026    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830f4875-c6ea-44d1-8d67-c38918d477a4-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.832037    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830f4875-c6ea-44d1-8d67-c38918d477a4-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.832067    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.832080    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2548cff-7724-4e35-a7a1-c97c40f7dc5f-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:38.832453 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:38.832095    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830f4875-c6ea-44d1-8d67-c38918d477a4-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:39.129402 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.129319    4837 scope.go:122] "RemoveContainer" containerID="ec9920632c659b64608b841cc31b75fd57c583af90934aacc882ec9fd7bd8e2e"
Nov 01 22:51:39.140331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.140293    4837 scope.go:122] "RemoveContainer" containerID="49055a0cd78bc9728445222731e0bed6f0c3884b8cceebbf1410c547cce8c3af"
Nov 01 22:51:39.150470 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.150454    4837 scope.go:122] "RemoveContainer" containerID="c7847ebc08a8c8b14f26a268c8efe486cd5bbe910ffc12fb6fcae7b767fffa67"
Nov 01 22:51:39.160342 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.160315    4837 scope.go:122] "RemoveContainer" containerID="b99eaed10cbda3cee75c9b8b649d77c54b7d5883d0ac6c55f3feaa39940f6d54"
Nov 01 22:51:39.169591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.169565    4837 scope.go:122] "RemoveContainer" containerID="42290030f6164fb3169243b25edefd856732539951637bf718ff94aa5a740378"
Nov 01 22:51:39.178906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.178881    4837 generic.go:356] "Generic (PLEG): container finished" podID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerID="2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f" exitCode=0
Nov 01 22:51:39.179084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.178929    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"
Nov 01 22:51:39.179084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.178962    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" event={"ID":"830f4875-c6ea-44d1-8d67-c38918d477a4","Type":"ContainerDied","Data":"2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f"}
Nov 01 22:51:39.179084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.179025    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj" event={"ID":"830f4875-c6ea-44d1-8d67-c38918d477a4","Type":"ContainerDied","Data":"ebf5ae74d5228843bc5f02808bde876c79b5b5250131e24489f8f7fbee494048"}
Nov 01 22:51:39.179084 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.179069    4837 scope.go:122] "RemoveContainer" containerID="2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f"
Nov 01 22:51:39.179586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.179565    4837 scope.go:122] "RemoveContainer" containerID="b332cc141ff3266f86ea5786c1f9e3c41d15a1c335b48dc9d860be2174b8c4e8"
Nov 01 22:51:39.187436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.187407    4837 generic.go:356] "Generic (PLEG): container finished" podID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerID="530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6" exitCode=0
Nov 01 22:51:39.187515 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.187462    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"
Nov 01 22:51:39.187515 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.187499    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" event={"ID":"e2548cff-7724-4e35-a7a1-c97c40f7dc5f","Type":"ContainerDied","Data":"530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6"}
Nov 01 22:51:39.187638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.187544    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w" event={"ID":"e2548cff-7724-4e35-a7a1-c97c40f7dc5f","Type":"ContainerDied","Data":"6233a986c1daaa3a9f92fef6ea74e9aff255c2ba86f8ff8924e23e6b672a3e1f"}
Nov 01 22:51:39.194331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.194192    4837 scope.go:122] "RemoveContainer" containerID="2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f"
Nov 01 22:51:39.194765 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:39.194736    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f\": container with ID starting with 2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f not found: ID does not exist" containerID="2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f"
Nov 01 22:51:39.194870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.194776    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f"} err="failed to get container status \"2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f\": rpc error: code = NotFound desc = could not find container \"2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f\": container with ID starting with 2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f not found: ID does not exist"
Nov 01 22:51:39.194870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.194800    4837 scope.go:122] "RemoveContainer" containerID="530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6"
Nov 01 22:51:39.197282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.196922    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n" event={"ID":"468199b3-71a0-4e45-927f-3dbe77075f56","Type":"ContainerDied","Data":"76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12"}
Nov 01 22:51:39.197282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.196993    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"
Nov 01 22:51:39.197789 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.197672    4837 scope.go:122] "RemoveContainer" containerID="2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f"
Nov 01 22:51:39.198158 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:39.198128    4837 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f\": rpc error: code = NotFound desc = could not find container \"2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f\": container with ID starting with 2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f not found: ID does not exist" containerID="2e56f18c0e0a556b7caf721da075ae0250da83cc109556d77fe3ba258bf4ba3f"
Nov 01 22:51:39.198263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.198161    4837 scope.go:122] "RemoveContainer" containerID="0bec5378d807383805db7d087406fd8fd21dc151164d953dc335da7c3a07b258"
Nov 01 22:51:39.200415 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.200398    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"
Nov 01 22:51:39.200494 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.200419    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx" event={"ID":"28eac005-a391-4ba5-b018-c870d18a1033","Type":"ContainerDied","Data":"26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1"}
Nov 01 22:51:39.213514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.213156    4837 scope.go:122] "RemoveContainer" containerID="530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6"
Nov 01 22:51:39.213514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.213234    4837 scope.go:122] "RemoveContainer" containerID="530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6"
Nov 01 22:51:39.215079 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:39.215020    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6\": container with ID starting with 530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6 not found: ID does not exist" containerID="530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6"
Nov 01 22:51:39.215202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.215093    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6"} err="failed to get container status \"530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6\": rpc error: code = NotFound desc = could not find container \"530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6\": container with ID starting with 530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6 not found: ID does not exist"
Nov 01 22:51:39.215202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.215117    4837 scope.go:122] "RemoveContainer" containerID="c385554dff7fdcb7278617177272e1ca4da51ce80898e3e371d340fadd4f4ec3"
Nov 01 22:51:39.215960 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:39.215931    4837 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6\": rpc error: code = NotFound desc = could not find container \"530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6\": container with ID starting with 530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6 not found: ID does not exist" containerID="530b01092deb65ff15cd774843ed4d463d0ad8585b74576a5dc339c5fc1e29b6"
Nov 01 22:51:39.216098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.215964    4837 scope.go:122] "RemoveContainer" containerID="f456f6dc6d3718f7845e90fd386a30f203824befd91b0eea26356581e481839d"
Nov 01 22:51:39.339929 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.339904    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"]
Nov 01 22:51:39.346446 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.346424    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gqdgx"]
Nov 01 22:51:39.464530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.464489    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"]
Nov 01 22:51:39.467568 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.467544    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-f2m5w"]
Nov 01 22:51:39.565208 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.565175    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"]
Nov 01 22:51:39.570944 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.570917    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-6pbfj"]
Nov 01 22:51:39.673714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.673682    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"]
Nov 01 22:51:39.679918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:39.679895    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-7w85n"]
Nov 01 22:51:40.416950 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:40.416921    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:51:40.528160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:40.528131    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28eac005-a391-4ba5-b018-c870d18a1033" path="/var/lib/kubelet/pods/28eac005-a391-4ba5-b018-c870d18a1033/volumes"
Nov 01 22:51:40.528742 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:40.528725    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" path="/var/lib/kubelet/pods/468199b3-71a0-4e45-927f-3dbe77075f56/volumes"
Nov 01 22:51:40.529354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:40.529335    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" path="/var/lib/kubelet/pods/830f4875-c6ea-44d1-8d67-c38918d477a4/volumes"
Nov 01 22:51:40.530244 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:40.530226    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" path="/var/lib/kubelet/pods/e2548cff-7724-4e35-a7a1-c97c40f7dc5f/volumes"
Nov 01 22:51:43.976496 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:43.976459    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 22:51:44.884974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.884927    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"]
Nov 01 22:51:44.885298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.885251    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" containerID="cri-o://0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6" gracePeriod=30
Nov 01 22:51:44.932377 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.932327    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"]
Nov 01 22:51:44.932609 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.932576    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy" containerID="cri-o://9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57" gracePeriod=30
Nov 01 22:51:44.937131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.937101    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"]
Nov 01 22:51:44.937384 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.937350    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy" containerID="cri-o://cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d" gracePeriod=30
Nov 01 22:51:44.978503 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.978469    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"]
Nov 01 22:51:44.978977 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.978701    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy" containerID="cri-o://692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f" gracePeriod=30
Nov 01 22:51:44.983316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.983286    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"]
Nov 01 22:51:44.983577 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:44.983546    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy" containerID="cri-o://7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707" gracePeriod=30
Nov 01 22:51:45.867335 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:45.867294    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache]"
Nov 01 22:51:47.356299 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:47.356263    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:47.356709 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:47.356323    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:48.814659 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:48.814626    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:51:49.021078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:49.020960    4837 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:49.021282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:49.021076    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:50.059383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.059346    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:51:50.059797 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.059413    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 01 22:51:50.248206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.248172    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:51:50.331022 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.330984    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:51:50.347305 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347270    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-podinfo\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.347478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347324    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-podinfo\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.347478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347387    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istiod-ca-cert\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.347478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347447    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-kube-api-access-d2nx9\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-kube-api-access-d2nx9\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.347646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347495    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-certs\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.347646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347520    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istiod-ca-cert\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.347646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347556    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-token\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-token\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.347646 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347590    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-socket\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.347834 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347649    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-certs\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.347834 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347727    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-token\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-token\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.347934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347844    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-envoy\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.347934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347898    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-data\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.348030 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.347980    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-envoy\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.348030 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.348010    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-credential-socket\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.348166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.348085    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-kube-api-access-x45tt\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-kube-api-access-x45tt\") pod \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\" (UID: \"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1\") "
Nov 01 22:51:50.348166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.348151    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-socket\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.348276 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.348204    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-data\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.348276 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.348241    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-credential-socket\") pod \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\" (UID: \"5d7eee6d-c6ff-440e-bade-d62562a6c57a\") "
Nov 01 22:51:50.353731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.349352    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-certs" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.353731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.349531    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-certs" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.353731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.349812    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istiod-ca-cert" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:50.353731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.350086    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-credential-socket" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.353731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.352397    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-socket" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.353731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.352777    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istiod-ca-cert" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:50.353731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.353103    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-credential-socket" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.353731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.353464    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-data" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.354692 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.354510    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-data" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.354692 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.354646    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-socket" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.367627 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.367569    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-token" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.367957 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.367916    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-envoy" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.395632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.395225    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-envoy" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.395632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.395284    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-token" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.395632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.395286    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-kube-api-access-d2nx9" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "kube-api-access-d2nx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.395632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.395314    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-kube-api-access-x45tt" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "kube-api-access-x45tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.402516 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.401800    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-podinfo" pod "5d7eee6d-c6ff-440e-bade-d62562a6c57a" (UID: "5d7eee6d-c6ff-440e-bade-d62562a6c57a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:50.403307 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.403271    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-podinfo" pod "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" (UID: "6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:50.415437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.415413    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:51:50.428474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.424352    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:51:50.428474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.426589    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450429    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-podinfo\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450474    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-token\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-token\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450517    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-data\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450554    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-certs\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450600    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-envoy\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450675    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-data\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450709    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-kube-api-access-l756p\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-kube-api-access-l756p\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450739    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-istio-token\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-istio-token\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.450689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450771    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-socket\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450825    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7578cf15-26e8-4b66-98e1-eb7af275381c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578cf15-26e8-4b66-98e1-eb7af275381c-istiod-ca-cert\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450888    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-credential-socket\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450917    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-podinfo\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.450948    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istiod-ca-cert\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451007    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-certs\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451094    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-socket\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451126    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44200b05-aec9-4756-9bc8-d50ec570847a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44200b05-aec9-4756-9bc8-d50ec570847a-istio-podinfo\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451157    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-envoy\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451235    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-credential-socket\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451266    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-kube-api-access-k8nnc\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-kube-api-access-k8nnc\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451331    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-token\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-token\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451433    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44200b05-aec9-4756-9bc8-d50ec570847a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44200b05-aec9-4756-9bc8-d50ec570847a-istiod-ca-cert\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451477    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-certs\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451550    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-credential-socket\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451600    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-kube-api-access-p6fnc\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-kube-api-access-p6fnc\") pod \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\" (UID: \"a3339144-9288-4e5d-b4f3-54b87e7dabb7\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451643    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-data\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451688    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-envoy\") pod \"7578cf15-26e8-4b66-98e1-eb7af275381c\" (UID: \"7578cf15-26e8-4b66-98e1-eb7af275381c\") "
Nov 01 22:51:50.451882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.451715    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-socket\") pod \"44200b05-aec9-4756-9bc8-d50ec570847a\" (UID: \"44200b05-aec9-4756-9bc8-d50ec570847a\") "
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452458    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452476    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452486    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452496    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452509    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452519    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x45tt\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-kube-api-access-x45tt\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452530    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452524    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istiod-ca-cert" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452539    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452577    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452601    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452615    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452631    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7eee6d-c6ff-440e-bade-d62562a6c57a-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452647    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2nx9\" (UniqueName: \"kubernetes.io/projected/5d7eee6d-c6ff-440e-bade-d62562a6c57a-kube-api-access-d2nx9\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452663    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7eee6d-c6ff-440e-bade-d62562a6c57a-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452677    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452691    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452705    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452719    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.452824 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452805    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-certs" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.453860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.452874    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44200b05-aec9-4756-9bc8-d50ec570847a-istiod-ca-cert" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:50.453860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.453013    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-socket" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.453860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.453140    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-certs" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.453860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.453327    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-credential-socket" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.453860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.453386    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-socket" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.453860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.453399    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-data" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.453860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.453691    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-credential-socket" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.454250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.454058    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-credential-socket" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.454250 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.454203    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7578cf15-26e8-4b66-98e1-eb7af275381c-istiod-ca-cert" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:50.455544 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.455518    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-certs" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.456029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.456004    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-data" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.456029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.456017    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44200b05-aec9-4756-9bc8-d50ec570847a-istio-podinfo" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:50.456235 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.456212    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-kube-api-access-p6fnc" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "kube-api-access-p6fnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.456303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.456287    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-socket" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.456372 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.456351    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-data" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.459089 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.459024    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-envoy" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.459169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.459053    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-podinfo" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:50.459169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.459037    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-token" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.460205 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.460184    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-envoy" pod "a3339144-9288-4e5d-b4f3-54b87e7dabb7" (UID: "a3339144-9288-4e5d-b4f3-54b87e7dabb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.460323 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.460305    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-token" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.460544 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.460519    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-envoy" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.460621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.460530    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-kube-api-access-l756p" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "kube-api-access-l756p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.460621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.460549    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-kube-api-access-k8nnc" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "kube-api-access-k8nnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.460621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.460529    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-podinfo" pod "7578cf15-26e8-4b66-98e1-eb7af275381c" (UID: "7578cf15-26e8-4b66-98e1-eb7af275381c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:50.461725 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.461707    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-istio-token" pod "44200b05-aec9-4756-9bc8-d50ec570847a" (UID: "44200b05-aec9-4756-9bc8-d50ec570847a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.496955 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.496921    4837 generic.go:356] "Generic (PLEG): container finished" podID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerID="692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f" exitCode=0
Nov 01 22:51:50.497170 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.496971    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"
Nov 01 22:51:50.497170 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.497009    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" event={"ID":"a3339144-9288-4e5d-b4f3-54b87e7dabb7","Type":"ContainerDied","Data":"692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f"}
Nov 01 22:51:50.497170 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.497109    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg" event={"ID":"a3339144-9288-4e5d-b4f3-54b87e7dabb7","Type":"ContainerDied","Data":"1e25a6a9c2c3f7726d1b354089002526507422109516bf9dbb4e22f18ab5861a"}
Nov 01 22:51:50.497170 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.497141    4837 scope.go:122] "RemoveContainer" containerID="692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f"
Nov 01 22:51:50.498783 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.498758    4837 generic.go:356] "Generic (PLEG): container finished" podID="44200b05-aec9-4756-9bc8-d50ec570847a" containerID="7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707" exitCode=0
Nov 01 22:51:50.498905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.498819    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"
Nov 01 22:51:50.498905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.498840    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" event={"ID":"44200b05-aec9-4756-9bc8-d50ec570847a","Type":"ContainerDied","Data":"7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707"}
Nov 01 22:51:50.498905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.498892    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz" event={"ID":"44200b05-aec9-4756-9bc8-d50ec570847a","Type":"ContainerDied","Data":"db4898f0b869a1011e35c163f270b5d5523f933497b3a43b68b3d917ea93684b"}
Nov 01 22:51:50.500795 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.500697    4837 generic.go:356] "Generic (PLEG): container finished" podID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerID="9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57" exitCode=0
Nov 01 22:51:50.500795 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.500722    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" event={"ID":"5d7eee6d-c6ff-440e-bade-d62562a6c57a","Type":"ContainerDied","Data":"9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57"}
Nov 01 22:51:50.500795 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.500754    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"
Nov 01 22:51:50.500795 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.500764    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp" event={"ID":"5d7eee6d-c6ff-440e-bade-d62562a6c57a","Type":"ContainerDied","Data":"704147f5b24c703ca40245488a099a5385a38263839a234731a95162259ddfe7"}
Nov 01 22:51:50.502700 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.502528    4837 generic.go:356] "Generic (PLEG): container finished" podID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerID="cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d" exitCode=0
Nov 01 22:51:50.502700 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.502613    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" event={"ID":"7578cf15-26e8-4b66-98e1-eb7af275381c","Type":"ContainerDied","Data":"cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d"}
Nov 01 22:51:50.502700 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.502655    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s" event={"ID":"7578cf15-26e8-4b66-98e1-eb7af275381c","Type":"ContainerDied","Data":"95a2946c2b0a292b7296b4a3f89c677d92aa5917c155390a6dda6f0770b6214b"}
Nov 01 22:51:50.503943 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.503814    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"
Nov 01 22:51:50.504603 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.504578    4837 generic.go:356] "Generic (PLEG): container finished" podID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerID="0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6" exitCode=0
Nov 01 22:51:50.505103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.505088    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"
Nov 01 22:51:50.505819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.505716    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" event={"ID":"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1","Type":"ContainerDied","Data":"0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6"}
Nov 01 22:51:50.505819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.505763    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z" event={"ID":"6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1","Type":"ContainerDied","Data":"fc78abbad0966eb5d368e91111b50b559d7a0485d870cf88e1ca2ea81375eaa5"}
Nov 01 22:51:50.520245 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.518379    4837 scope.go:122] "RemoveContainer" containerID="692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f"
Nov 01 22:51:50.522361 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:50.522334    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f\": container with ID starting with 692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f not found: ID does not exist" containerID="692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f"
Nov 01 22:51:50.522511 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.522488    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f"} err="failed to get container status \"692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f\": rpc error: code = NotFound desc = could not find container \"692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f\": container with ID starting with 692be0904f62e3de32ce0e9547b7f8f84b512726f186d1b3d74c8d8a1873538f not found: ID does not exist"
Nov 01 22:51:50.522615 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.522605    4837 scope.go:122] "RemoveContainer" containerID="7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707"
Nov 01 22:51:50.554146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554121    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554151    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l756p\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-kube-api-access-l756p\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554169    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554183    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554196    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578cf15-26e8-4b66-98e1-eb7af275381c-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554209    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554223    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554235    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554248    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554261    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554274    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44200b05-aec9-4756-9bc8-d50ec570847a-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554287    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554299    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554314    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8nnc\" (UniqueName: \"kubernetes.io/projected/44200b05-aec9-4756-9bc8-d50ec570847a-kube-api-access-k8nnc\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554327    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554339    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44200b05-aec9-4756-9bc8-d50ec570847a-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554352    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554366    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554378    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fnc\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-kube-api-access-p6fnc\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554390    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554403    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554416    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554428    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578cf15-26e8-4b66-98e1-eb7af275381c-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554440    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554452    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3339144-9288-4e5d-b4f3-54b87e7dabb7-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554465    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.554483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.554478    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44200b05-aec9-4756-9bc8-d50ec570847a-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.557651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.557627    4837 scope.go:122] "RemoveContainer" containerID="7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707"
Nov 01 22:51:50.558152 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:50.558125    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707\": container with ID starting with 7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707 not found: ID does not exist" containerID="7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707"
Nov 01 22:51:50.558325 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.558293    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707"} err="failed to get container status \"7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707\": rpc error: code = NotFound desc = could not find container \"7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707\": container with ID starting with 7a33388c5ced341587b8a9a614b02dbcc5bb575c899f1c4f10fbd53af710d707 not found: ID does not exist"
Nov 01 22:51:50.558437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.558418    4837 scope.go:122] "RemoveContainer" containerID="9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57"
Nov 01 22:51:50.568921 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.568903    4837 scope.go:122] "RemoveContainer" containerID="9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57"
Nov 01 22:51:50.572806 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:50.572776    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57\": container with ID starting with 9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57 not found: ID does not exist" containerID="9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57"
Nov 01 22:51:50.572905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.572816    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57"} err="failed to get container status \"9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57\": rpc error: code = NotFound desc = could not find container \"9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57\": container with ID starting with 9cb8dc10bec31533fd1dd1a9be7f742b88491d5969a7b075d1ab926097fd7a57 not found: ID does not exist"
Nov 01 22:51:50.572905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.572846    4837 scope.go:122] "RemoveContainer" containerID="cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d"
Nov 01 22:51:50.584871 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.584851    4837 scope.go:122] "RemoveContainer" containerID="cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d"
Nov 01 22:51:50.585151 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:50.585130    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d\": container with ID starting with cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d not found: ID does not exist" containerID="cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d"
Nov 01 22:51:50.585219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.585164    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d"} err="failed to get container status \"cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d\": rpc error: code = NotFound desc = could not find container \"cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d\": container with ID starting with cb30a4fb1da074ee8c769604145bc27b9f22140dec1b8ed2ef656f882d286b6d not found: ID does not exist"
Nov 01 22:51:50.585219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.585185    4837 scope.go:122] "RemoveContainer" containerID="0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6"
Nov 01 22:51:50.596118 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.596104    4837 scope.go:122] "RemoveContainer" containerID="0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6"
Nov 01 22:51:50.596409 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:50.596380    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6\": container with ID starting with 0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6 not found: ID does not exist" containerID="0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6"
Nov 01 22:51:50.596511 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:50.596418    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6"} err="failed to get container status \"0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6\": rpc error: code = NotFound desc = could not find container \"0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6\": container with ID starting with 0002bd641ad9519e5a7e9a861b6aae5cef59dd443bee265bb12a276dbfe0a3b6 not found: ID does not exist"
Nov 01 22:51:51.115231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.115202    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"]
Nov 01 22:51:51.118966 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.118941    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-dv4zp"]
Nov 01 22:51:51.221190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.221109    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"]
Nov 01 22:51:51.227995 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.227963    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-bv7dz"]
Nov 01 22:51:51.329192 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.329160    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"]
Nov 01 22:51:51.335146 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.335119    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x4cpg"]
Nov 01 22:51:51.436425 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.436390    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"]
Nov 01 22:51:51.440220 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.440191    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-x545s"]
Nov 01 22:51:51.542314 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.542223    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"]
Nov 01 22:51:51.546221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:51.546195    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-cnk6z"]
Nov 01 22:51:52.529234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:52.529203    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" path="/var/lib/kubelet/pods/44200b05-aec9-4756-9bc8-d50ec570847a/volumes"
Nov 01 22:51:52.529746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:52.529730    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" path="/var/lib/kubelet/pods/5d7eee6d-c6ff-440e-bade-d62562a6c57a/volumes"
Nov 01 22:51:52.530215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:52.530201    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" path="/var/lib/kubelet/pods/6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1/volumes"
Nov 01 22:51:52.530963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:52.530949    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" path="/var/lib/kubelet/pods/7578cf15-26e8-4b66-98e1-eb7af275381c/volumes"
Nov 01 22:51:52.531418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:51:52.531395    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" path="/var/lib/kubelet/pods/a3339144-9288-4e5d-b4f3-54b87e7dabb7/volumes"
Nov 01 22:51:55.953065 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:55.952989    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache]"
Nov 01 22:51:58.923300 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:51:58.923253    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:06.032359 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:06.032319    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:13.965998 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:13.965960    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:16.046486 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:16.046444    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:18.815502 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:18.815469    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:52:26.158578 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:26.158532    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:28.151374 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:28.151333    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:28.151750 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:28.151398    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:28.918340 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:28.918302    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:36.180615 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:36.180573    4837 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice/crio-26d4f9ff68889ab9a6a570a49ab05f5c38670f69a4922ce8289bb216759c5ec1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468199b3_71a0_4e45_927f_3dbe77075f56.slice/crio-76651af229158ccec4f908208e10fbb3ae28ea587060fc544d3e39d653524b12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28eac005_a391_4ba5_b018_c870d18a1033.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 22:52:38.273726 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.273692    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"]
Nov 01 22:52:38.274612 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274594    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy"
Nov 01 22:52:38.274674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274614    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy"
Nov 01 22:52:38.274674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274628    4837 cpu_manager.go:414] "containerMap: removing container" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy"
Nov 01 22:52:38.274674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274633    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy"
Nov 01 22:52:38.274674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274647    4837 cpu_manager.go:414] "containerMap: removing container" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy"
Nov 01 22:52:38.274674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274652    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy"
Nov 01 22:52:38.274674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274665    4837 cpu_manager.go:414] "containerMap: removing container" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy"
Nov 01 22:52:38.274674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274669    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274682    4837 cpu_manager.go:414] "containerMap: removing container" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274686    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274714    4837 cpu_manager.go:414] "containerMap: removing container" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274719    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274728    4837 cpu_manager.go:414] "containerMap: removing container" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274732    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274746    4837 cpu_manager.go:414] "containerMap: removing container" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274751    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274759    4837 cpu_manager.go:414] "containerMap: removing container" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274763    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274904    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2548cff-7724-4e35-a7a1-c97c40f7dc5f" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274914    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="44200b05-aec9-4756-9bc8-d50ec570847a" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274921    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="468199b3-71a0-4e45-927f-3dbe77075f56" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274929    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7eee6d-c6ff-440e-bade-d62562a6c57a" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274941    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="28eac005-a391-4ba5-b018-c870d18a1033" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274953    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7119d22b-44f1-4c43-8a7b-c1dc954edaab" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274961    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="830f4875-c6ea-44d1-8d67-c38918d477a4" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274970    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3339144-9288-4e5d-b4f3-54b87e7dabb7" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274987    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7578cf15-26e8-4b66-98e1-eb7af275381c" containerName="istio-proxy"
Nov 01 22:52:38.274999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.274995    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa38a5f-c2bc-46fb-999a-cf6ab6f3a4d1" containerName="istio-proxy"
Nov 01 22:52:38.280983 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.280961    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311291 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311258    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-istio-token\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311300    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e21ad7-e679-4381-a949-814c8b5207cd-istio-podinfo\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311472    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e21ad7-e679-4381-a949-814c8b5207cd-istiod-ca-cert\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311575 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311559    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-data\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311622 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311607    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-credential-socket\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311718 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311707    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-socket\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311762 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311726    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tx9\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-kube-api-access-f9tx9\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311881    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-certs\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.311937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.311912    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-envoy\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.329997 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.329968    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-f5b935-openshift-default-dockercfg-d8k54\""
Nov 01 22:52:38.380282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.380249    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"]
Nov 01 22:52:38.413480 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.413451    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-socket\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.413704 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.413684    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tx9\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-kube-api-access-f9tx9\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.413899 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.413872    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-socket\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.414115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.414079    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-certs\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.414285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.414145    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-envoy\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.414285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.414182    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-istio-token\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.414285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.414212    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e21ad7-e679-4381-a949-814c8b5207cd-istio-podinfo\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.414285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.414275    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e21ad7-e679-4381-a949-814c8b5207cd-istiod-ca-cert\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.414666 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.414326    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-data\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.414666 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.414363    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-credential-socket\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.414840 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.414819    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-credential-socket\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.415711 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.415690    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-certs\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.556108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.459502    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-data\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.556108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.460588    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e21ad7-e679-4381-a949-814c8b5207cd-istiod-ca-cert\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.556108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.472400    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-envoy\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.556108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.532242    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tx9\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-kube-api-access-f9tx9\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.556108 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.546555    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-istio-token\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.654134 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:38.650032    4837 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/a50933162e76a1c9f9502bdd16975b16f5ba263618bc5c387388198c4cb0d0f9/diff" to get inode usage: stat /var/lib/containers/storage/overlay/a50933162e76a1c9f9502bdd16975b16f5ba263618bc5c387388198c4cb0d0f9/diff: no such file or directory, extraDiskErr: <nil>
Nov 01 22:52:38.654134 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:38.650117    4837 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/23d832f8f86da90963ddf3545d96dde3bef702edd61b06316571febe8f82a80c/diff" to get inode usage: stat /var/lib/containers/storage/overlay/23d832f8f86da90963ddf3545d96dde3bef702edd61b06316571febe8f82a80c/diff: no such file or directory, extraDiskErr: <nil>
Nov 01 22:52:38.687169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.687131    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e21ad7-e679-4381-a949-814c8b5207cd-istio-podinfo\") pod \"e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") " pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:38.952392 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.952366    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-f5b935-openshift-default-dockercfg-d8k54\""
Nov 01 22:52:38.955402 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:38.955387    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:39.342505 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:39.342434    4837 scope.go:122] "RemoveContainer" containerID="f444ae0f3c20f38d20272cd3b8c4f58edf0624de7ef7e2c8163211010ece614b"
Nov 01 22:52:39.352492 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:39.352473    4837 scope.go:122] "RemoveContainer" containerID="4d94fed2561ef71d7e17345727083e45733d9d6d176ac59db7278461061118da"
Nov 01 22:52:39.360987 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:39.360966    4837 scope.go:122] "RemoveContainer" containerID="adc9e63c6c278416eb8489bc6161dfddc0b4984ed91ad6f337a0ab192a63f25a"
Nov 01 22:52:39.385211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:39.385193    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"]
Nov 01 22:52:39.411098 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:52:39.411074    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e21ad7_e679_4381_a949_814c8b5207cd.slice/crio-3cec13261211e16c1f886b7ae4baf9939dc055581d8efd73a73bfa3daf5ed710 WatchSource:0}: Error finding container 3cec13261211e16c1f886b7ae4baf9939dc055581d8efd73a73bfa3daf5ed710: Status 404 returned error can't find the container with id 3cec13261211e16c1f886b7ae4baf9939dc055581d8efd73a73bfa3daf5ed710
Nov 01 22:52:39.414678 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:39.414648    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:52:39.414730 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:39.414714    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:52:39.414768 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:39.414742    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:52:40.153086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:40.147731    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:52:40.232916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:40.232859    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" event={"ID":"80e21ad7-e679-4381-a949-814c8b5207cd","Type":"ContainerStarted","Data":"ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f"}
Nov 01 22:52:40.233100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:40.232942    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" event={"ID":"80e21ad7-e679-4381-a949-814c8b5207cd","Type":"ContainerStarted","Data":"3cec13261211e16c1f886b7ae4baf9939dc055581d8efd73a73bfa3daf5ed710"}
Nov 01 22:52:40.377393 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:40.377331    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" podStartSLOduration=2.377312174 podStartE2EDuration="2.377312174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:40.372719174 +0000 UTC m=+1741.961918233" watchObservedRunningTime="2026-11-01 22:52:40.377312174 +0000 UTC m=+1741.966511236"
Nov 01 22:52:40.961403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:40.961373    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:40.972210 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:40.972184    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:41.263353 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:41.263283    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:41.271065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:41.266966    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:52:49.127289 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:49.123242    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:52:49.708875 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:49.708834    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"]
Nov 01 22:52:49.709164 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:49.709136    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8" podUID="bcb12256-dfbe-401b-91b5-ee1f9a2106cd" containerName="maas-api" containerID="cri-o://44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713" gracePeriod=30
Nov 01 22:52:50.116795 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.116771    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:52:50.220712 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.220683    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-maas-api-tls\") pod \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") "
Nov 01 22:52:50.221215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.220737    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-kube-api-access-29mmb\" (UniqueName: \"kubernetes.io/projected/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-kube-api-access-29mmb\") pod \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") "
Nov 01 22:52:50.221215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.220839    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-openshift-service-ca\") pod \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") "
Nov 01 22:52:50.221215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.220961    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-metrics-certs\") pod \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\" (UID: \"bcb12256-dfbe-401b-91b5-ee1f9a2106cd\") "
Nov 01 22:52:50.221358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.221226    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-openshift-service-ca" pod "bcb12256-dfbe-401b-91b5-ee1f9a2106cd" (UID: "bcb12256-dfbe-401b-91b5-ee1f9a2106cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:50.221747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.221723    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:50.223120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.223092    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-kube-api-access-29mmb" pod "bcb12256-dfbe-401b-91b5-ee1f9a2106cd" (UID: "bcb12256-dfbe-401b-91b5-ee1f9a2106cd"). InnerVolumeSpecName "kube-api-access-29mmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:50.223120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.223117    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-maas-api-tls" pod "bcb12256-dfbe-401b-91b5-ee1f9a2106cd" (UID: "bcb12256-dfbe-401b-91b5-ee1f9a2106cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:52:50.223248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.223138    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-metrics-certs" pod "bcb12256-dfbe-401b-91b5-ee1f9a2106cd" (UID: "bcb12256-dfbe-401b-91b5-ee1f9a2106cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:52:50.333163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.328922    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:50.333401 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.333378    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:50.356531 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.356482    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29mmb\" (UniqueName: \"kubernetes.io/projected/bcb12256-dfbe-401b-91b5-ee1f9a2106cd-kube-api-access-29mmb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:50.510068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.509618    4837 generic.go:356] "Generic (PLEG): container finished" podID="bcb12256-dfbe-401b-91b5-ee1f9a2106cd" containerID="44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713" exitCode=0
Nov 01 22:52:50.510068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.509896    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8" event={"ID":"bcb12256-dfbe-401b-91b5-ee1f9a2106cd","Type":"ContainerDied","Data":"44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713"}
Nov 01 22:52:50.510068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.510000    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8" event={"ID":"bcb12256-dfbe-401b-91b5-ee1f9a2106cd","Type":"ContainerDied","Data":"dbae5e37695dccfeb8c6861389e4c27af453c4195672a6dc4c65806607b8a791"}
Nov 01 22:52:50.512144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.510406    4837 scope.go:122] "RemoveContainer" containerID="44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713"
Nov 01 22:52:50.512144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.510609    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"
Nov 01 22:52:50.611745 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.611714    4837 scope.go:122] "RemoveContainer" containerID="44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713"
Nov 01 22:52:50.614011 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:52:50.613981    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713\": container with ID starting with 44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713 not found: ID does not exist" containerID="44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713"
Nov 01 22:52:50.614327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.614022    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713"} err="failed to get container status \"44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713\": rpc error: code = NotFound desc = could not find container \"44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713\": container with ID starting with 44e376bd9a4be99bdc9815091052dfda0ea9136de72e4aed7a0564408af84713 not found: ID does not exist"
Nov 01 22:52:50.742143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.733093    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"]
Nov 01 22:52:50.762126 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:50.762078    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de3d6ade-9464fdd44-8ncl8"]
Nov 01 22:52:52.526854 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:52.526817    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb12256-dfbe-401b-91b5-ee1f9a2106cd" path="/var/lib/kubelet/pods/bcb12256-dfbe-401b-91b5-ee1f9a2106cd/volumes"
Nov 01 22:52:52.528979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:52.528956    4837 generic.go:356] "Generic (PLEG): container finished" podID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerID="f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363" exitCode=0
Nov 01 22:52:52.529149 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:52.529000    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" event={"ID":"2baff4f5-c9e3-4860-9a29-2b36a1e09547","Type":"ContainerDied","Data":"f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363"}
Nov 01 22:52:52.529327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:52.529313    4837 scope.go:122] "RemoveContainer" containerID="f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363"
Nov 01 22:52:52.531166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:52.531128    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:52:52.531278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:52.531215    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:52:52.531278 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:52.531260    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:52:53.558073 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:53.557590    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" event={"ID":"2baff4f5-c9e3-4860-9a29-2b36a1e09547","Type":"ContainerStarted","Data":"fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b"}
Nov 01 22:52:53.853131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:53.841813    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:52:53.853131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:53.843878    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 01 22:52:53.853131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:53.844103    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 01 22:52:54.852065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:54.845335    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 01 22:52:54.852065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:54.845406    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 01 22:52:55.619597 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:55.619565    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:52:55.853066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:55.852335    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 01 22:52:55.853066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:55.852404    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 01 22:52:56.847067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:56.841435    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 01 22:52:56.847067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:56.841512    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 01 22:52:57.856169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:57.856128    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 01 22:52:57.856694 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:57.856187    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 01 22:52:58.529969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:58.529940    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"]
Nov 01 22:52:58.530204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:58.530151    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy" containerID="cri-o://570655f006461514baf5c3eba7644022df3430c19b26d3ad1d95f256b6d0de39" gracePeriod=30
Nov 01 22:52:58.847124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:58.846534    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 01 22:52:58.847124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:58.846590    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 01 22:52:59.845704 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:59.845673    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:52:59.846724 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:52:59.846706    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 22:53:03.705522 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.705491    4837 generic.go:356] "Generic (PLEG): container finished" podID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerID="570655f006461514baf5c3eba7644022df3430c19b26d3ad1d95f256b6d0de39" exitCode=0
Nov 01 22:53:03.705881 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.705562    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" event={"ID":"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7","Type":"ContainerDied","Data":"570655f006461514baf5c3eba7644022df3430c19b26d3ad1d95f256b6d0de39"}
Nov 01 22:53:03.705881 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.705622    4837 scope.go:122] "RemoveContainer" containerID="055aaae705f7618bdd13077ab1766012f0de79cf7f423f65bfc26c20d02d2677"
Nov 01 22:53:03.855255 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.855233    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:53:03.946444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946419    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-credential-socket\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.946588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946546    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-podinfo\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.946663 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946587    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istiod-ca-cert\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.946663 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946637    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-kube-api-access-6xfs2\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-kube-api-access-6xfs2\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.946770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946680    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-certs\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.946770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946692    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-credential-socket" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:53:03.946770 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946710    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-token\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-token\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.946922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946771    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-socket\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.946922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946864    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-data\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.946922 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946893    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-envoy\") pod \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\" (UID: \"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7\") "
Nov 01 22:53:03.947102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.946961    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-certs" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:53:03.947102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.947014    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istiod-ca-cert" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:53:03.947242 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.947216    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-socket" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:53:03.947381 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.947359    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-data" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:53:03.947724 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.947699    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:03.947823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.947728    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:03.947823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.947744    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:03.947823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.947758    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:03.947823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.947773    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:03.949022 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.948993    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-podinfo" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:53:03.949195 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.949173    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-token" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:53:03.949264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.949231    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-kube-api-access-6xfs2" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "kube-api-access-6xfs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:53:03.949328 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:03.949295    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-envoy" pod "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" (UID: "26f1a6f7-57be-4ca6-86e0-a45714f3f7b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:53:04.049412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.049379    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:04.049412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.049405    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:04.049412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.049414    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:04.049674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.049424    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfs2\" (UniqueName: \"kubernetes.io/projected/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7-kube-api-access-6xfs2\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:04.713328 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.713294    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"
Nov 01 22:53:04.713328 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.713300    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs" event={"ID":"26f1a6f7-57be-4ca6-86e0-a45714f3f7b7","Type":"ContainerDied","Data":"f77e913fbf3039030fcc6e2bf4dbbc50f61dcf1c4d4b98b2cb84916c417902c6"}
Nov 01 22:53:04.713864 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.713366    4837 scope.go:122] "RemoveContainer" containerID="570655f006461514baf5c3eba7644022df3430c19b26d3ad1d95f256b6d0de39"
Nov 01 22:53:04.917541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.917513    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"]
Nov 01 22:53:04.963457 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:04.963435    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-de3d6ade-openshift-default-d6b7f5f97-4ccfs"]
Nov 01 22:53:06.526055 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:06.526010    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" path="/var/lib/kubelet/pods/26f1a6f7-57be-4ca6-86e0-a45714f3f7b7/volumes"
Nov 01 22:53:13.583835 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:13.583791    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:53:18.813940 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:53:18.813904    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:53:24.213183 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:24.213148    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:53:24.213580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:24.213216    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:53:24.924000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:24.923971    4837 generic.go:356] "Generic (PLEG): container finished" podID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerID="56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c" exitCode=0
Nov 01 22:53:24.924202 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:24.924032    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" event={"ID":"6b56f6a2-6dad-405e-9d88-c30f0f9cee77","Type":"ContainerDied","Data":"56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c"}
Nov 01 22:53:24.924444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:24.924427    4837 scope.go:122] "RemoveContainer" containerID="56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c"
Nov 01 22:53:24.925985 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:24.925961    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:24.926079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:24.926025    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:24.926135 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:24.926079    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:25.985062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:25.983847    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" event={"ID":"6b56f6a2-6dad-405e-9d88-c30f0f9cee77","Type":"ContainerStarted","Data":"c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7"}
Nov 01 22:53:26.155074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:26.154423    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:53:26.155074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:26.154542    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 22:53:26.155074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:26.154582    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 22:53:27.154068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:27.150968    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 22:53:27.154068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:27.151029    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 22:53:28.147465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:28.147425    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 22:53:28.147741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:28.147712    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 22:53:29.155221 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:29.155173    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 22:53:29.155663 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:29.155249    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 22:53:30.148066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:30.146425    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 01 22:53:30.148066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:30.146486    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 01 22:53:31.146616 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:31.146585    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:53:31.147022 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:31.146954    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:53:31.147732 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:31.147711    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 22:53:35.138441 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.138408    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"]
Nov 01 22:53:35.139098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139082    4837 cpu_manager.go:414] "containerMap: removing container" podUID="bcb12256-dfbe-401b-91b5-ee1f9a2106cd" containerName="maas-api"
Nov 01 22:53:35.139160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139098    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb12256-dfbe-401b-91b5-ee1f9a2106cd" containerName="maas-api"
Nov 01 22:53:35.139160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139124    4837 cpu_manager.go:414] "containerMap: removing container" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy"
Nov 01 22:53:35.139160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139128    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy"
Nov 01 22:53:35.139160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139137    4837 cpu_manager.go:414] "containerMap: removing container" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy"
Nov 01 22:53:35.139160 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139142    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy"
Nov 01 22:53:35.139306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139258    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb12256-dfbe-401b-91b5-ee1f9a2106cd" containerName="maas-api"
Nov 01 22:53:35.139306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139268    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy"
Nov 01 22:53:35.139306 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.139273    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f1a6f7-57be-4ca6-86e0-a45714f3f7b7" containerName="istio-proxy"
Nov 01 22:53:35.142598 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.142583    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.191670 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.191647    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bd7ff8-serving-cert\""
Nov 01 22:53:35.244285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.244253    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"]
Nov 01 22:53:35.252063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.252017    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.252196 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.252132    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.252255 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.252196    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44tm6\" (UniqueName: \"kubernetes.io/projected/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-kube-api-access-44tm6\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.252390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.252375    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-metrics-certs\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.355506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.355472    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-metrics-certs\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.355817 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.355795    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.355967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.355952    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.356158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.356140    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44tm6\" (UniqueName: \"kubernetes.io/projected/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-kube-api-access-44tm6\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.363966 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:53:35.363933    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-serving-cert: secret "maas-api-e2e-api-a-bd7ff8-serving-cert" not found
Nov 01 22:53:35.364237 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:53:35.364219    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls podName:7fbadf4f-8515-48eb-bff3-6a993d35e3a4 nodeName:}" failed. No retries permitted until 2026-11-01 22:53:35.864194971 +0000 UTC m=+1797.453394021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls") pod "maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq" (UID: "7fbadf4f-8515-48eb-bff3-6a993d35e3a4") : secret "maas-api-e2e-api-a-bd7ff8-serving-cert" not found
Nov 01 22:53:35.364711 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.364689    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-metrics-certs\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.364825 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.364803    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.423661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.417650    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44tm6\" (UniqueName: \"kubernetes.io/projected/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-kube-api-access-44tm6\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.876511 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.876431    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:35.878861 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:35.878841    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls\") pod \"maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:36.055589 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:36.055559    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:36.430198 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:36.430173    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:53:36.431062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:36.431022    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"]
Nov 01 22:53:37.057687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:37.057654    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq" event={"ID":"7fbadf4f-8515-48eb-bff3-6a993d35e3a4","Type":"ContainerStarted","Data":"e54c247a197b0d24cade06c6d9e882b501c8f9f0679197173bc6b22390714a9b"}
Nov 01 22:53:38.066547 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:38.066513    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq" event={"ID":"7fbadf4f-8515-48eb-bff3-6a993d35e3a4","Type":"ContainerStarted","Data":"abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1"}
Nov 01 22:53:38.066914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:38.066582    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:38.174604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:38.174553    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq" podStartSLOduration=2.248150737 podStartE2EDuration="3.174535993s" totalImagesPullingTime="926.385256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:38.174418522 +0000 UTC m=+1799.763617582" watchObservedRunningTime="2026-11-01 22:53:38.174535993 +0000 UTC m=+1799.763735052"
Nov 01 22:53:38.758894 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:38.758861    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:53:38.765027 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:38.765002    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:53:40.156065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.153725    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"]
Nov 01 22:53:40.346064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.344204    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.368863 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.368828    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"]
Nov 01 22:53:40.384058 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384018    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-envoy\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.384232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384210    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-socket\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.384316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384301    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-podinfo\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.384354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384333    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-data\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.384392 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384368    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4bbc\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-kube-api-access-t4bbc\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.384427 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384414    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-credential-socket\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.384523 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384509    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0049fa-0562-4103-8df1-ba15d48f58ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.384614 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384601    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-certs\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.384683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.384670    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-token\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.398931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.398899    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-8cc47a-openshift-default-dockercfg-4jzs8\""
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.541638    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-certs\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.541752    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-token\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.541835    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-envoy\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.541936    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-socket\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.542014    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-podinfo\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.542063    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-data\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.542099    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4bbc\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-kube-api-access-t4bbc\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.542158    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-credential-socket\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.542253    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0049fa-0562-4103-8df1-ba15d48f58ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.542632    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-certs\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.542965    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-socket\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.543689 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.543156    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0049fa-0562-4103-8df1-ba15d48f58ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.559947 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.551408    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-data\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.559947 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.551629    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-credential-socket\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.559947 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.554190    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-podinfo\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.559947 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.558259    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-envoy\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.609067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.607981    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-token\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.613330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.613298    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4bbc\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-kube-api-access-t4bbc\") pod \"e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") " pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:40.663495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:40.663471    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:41.090065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:41.087798    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"]
Nov 01 22:53:41.103617 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:41.103578    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:41.103736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:41.103662    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:41.103736 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:41.103707    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:41.323145 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:41.323111    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" event={"ID":"cf0049fa-0562-4103-8df1-ba15d48f58ba","Type":"ContainerStarted","Data":"6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954"}
Nov 01 22:53:41.323604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:41.323168    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" event={"ID":"cf0049fa-0562-4103-8df1-ba15d48f58ba","Type":"ContainerStarted","Data":"a17ce1b5510087a6bcb927140f21204f8ca43a81eb429d5eee65c11b15df9e8e"}
Nov 01 22:53:41.436507 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:41.436230    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" podStartSLOduration=1.436217252 podStartE2EDuration="1.436217252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:41.435210783 +0000 UTC m=+1803.024409843" watchObservedRunningTime="2026-11-01 22:53:41.436217252 +0000 UTC m=+1803.025416312"
Nov 01 22:53:41.665339 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:41.665153    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:42.670103 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:42.670030    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:43.338016 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:43.337987    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:43.338969 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:43.338952    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:53:44.079822 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:44.079798    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 22:53:48.359237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.347424    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"]
Nov 01 22:53:48.361316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.361030    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:48.369131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.368935    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"]
Nov 01 22:53:48.411777 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.411753    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8cc47a\""
Nov 01 22:53:48.411777 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.411758    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8cc47a-dockercfg-8gq6d\""
Nov 01 22:53:48.473649 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.473619    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t\" (UID: \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:48.473934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.473914    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn2sw\" (UniqueName: \"kubernetes.io/projected/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-kube-api-access-vn2sw\") pod \"payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t\" (UID: \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:48.575741 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.575712    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2sw\" (UniqueName: \"kubernetes.io/projected/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-kube-api-access-vn2sw\") pod \"payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t\" (UID: \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:48.575895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.575818    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t\" (UID: \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:48.576479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.576459    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t\" (UID: \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:48.630588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.630535    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2sw\" (UniqueName: \"kubernetes.io/projected/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-kube-api-access-vn2sw\") pod \"payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t\" (UID: \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:48.689226 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.689206    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:48.755948 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.755918    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7137-2f1e1b"]
Nov 01 22:53:48.769697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.766983    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:48.812174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.812140    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7137-2f1e1b"]
Nov 01 22:53:48.816464 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.816435    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5sq6\""
Nov 01 22:53:48.820093 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:53:48.820061    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:53:48.881964 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.881899    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9km5t\" (UniqueName: \"kubernetes.io/projected/dbbcc5c2-024d-460f-bd94-9ba4aba17eb9-kube-api-access-9km5t\") pod \"test-curl-7137-2f1e1b\" (UID: \"dbbcc5c2-024d-460f-bd94-9ba4aba17eb9\") " pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:48.983425 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:48.983399    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9km5t\" (UniqueName: \"kubernetes.io/projected/dbbcc5c2-024d-460f-bd94-9ba4aba17eb9-kube-api-access-9km5t\") pod \"test-curl-7137-2f1e1b\" (UID: \"dbbcc5c2-024d-460f-bd94-9ba4aba17eb9\") " pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:49.036476 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:49.036454    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9km5t\" (UniqueName: \"kubernetes.io/projected/dbbcc5c2-024d-460f-bd94-9ba4aba17eb9-kube-api-access-9km5t\") pod \"test-curl-7137-2f1e1b\" (UID: \"dbbcc5c2-024d-460f-bd94-9ba4aba17eb9\") " pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:49.067089 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:53:49.067063    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4aa13c_e4dc_48ad_ab95_127ab1a6bb14.slice/crio-d44500fadf239d8c095ec2a3e2c013f1b37eb0364ed64bc093c2e6cc57bc591f WatchSource:0}: Error finding container d44500fadf239d8c095ec2a3e2c013f1b37eb0364ed64bc093c2e6cc57bc591f: Status 404 returned error can't find the container with id d44500fadf239d8c095ec2a3e2c013f1b37eb0364ed64bc093c2e6cc57bc591f
Nov 01 22:53:49.067528 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:49.067458    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"]
Nov 01 22:53:49.086237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:49.086219    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:49.408349 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:49.408266    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t" event={"ID":"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14","Type":"ContainerStarted","Data":"b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210"}
Nov 01 22:53:49.408349 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:49.408326    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t" event={"ID":"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14","Type":"ContainerStarted","Data":"d44500fadf239d8c095ec2a3e2c013f1b37eb0364ed64bc093c2e6cc57bc591f"}
Nov 01 22:53:49.408734 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:49.408357    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:53:49.456166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:49.456142    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7137-2f1e1b"]
Nov 01 22:53:49.456324 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:53:49.456304    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbbcc5c2_024d_460f_bd94_9ba4aba17eb9.slice/crio-80022097814a9c4723f63b63bbd95c0af7f30a2bb19e5133b93a850c60988258 WatchSource:0}: Error finding container 80022097814a9c4723f63b63bbd95c0af7f30a2bb19e5133b93a850c60988258: Status 404 returned error can't find the container with id 80022097814a9c4723f63b63bbd95c0af7f30a2bb19e5133b93a850c60988258
Nov 01 22:53:49.516829 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:49.516787    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t" podStartSLOduration=1.516774228 podStartE2EDuration="1.516774228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:49.514205373 +0000 UTC m=+1811.103404436" watchObservedRunningTime="2026-11-01 22:53:49.516774228 +0000 UTC m=+1811.105973287"
Nov 01 22:53:50.417946 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:50.417907    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7137-2f1e1b" event={"ID":"dbbcc5c2-024d-460f-bd94-9ba4aba17eb9","Type":"ContainerStarted","Data":"80022097814a9c4723f63b63bbd95c0af7f30a2bb19e5133b93a850c60988258"}
Nov 01 22:53:51.427489 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:51.427453    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7137-2f1e1b" event={"ID":"dbbcc5c2-024d-460f-bd94-9ba4aba17eb9","Type":"ContainerStarted","Data":"0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c"}
Nov 01 22:53:51.559167 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:51.559107    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7137-2f1e1b" podStartSLOduration=2.024809644 podStartE2EDuration="3.559089292s" totalImagesPullingTime="1.534279648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:51.556375577 +0000 UTC m=+1813.145574638" watchObservedRunningTime="2026-11-01 22:53:51.559089292 +0000 UTC m=+1813.148288351"
Nov 01 22:53:53.066263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:53.066223    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7137-2f1e1b"]
Nov 01 22:53:53.070556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:53.070531    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7137-2f1e1b"]
Nov 01 22:53:53.466484 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:53.466422    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7137-2f1e1b" podUID="dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" containerName="test-curl-7137-2f1e1b" containerID="cri-o://0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c" gracePeriod=2
Nov 01 22:53:53.514336 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:53:53.514293    4837 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-2f1e1b\" is forbidden: User \"system:node:ip-10-0-129-77.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-77.us-west-2.compute.internal' and this object" podUID="dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:55.864731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:55.864705    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:55.918128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:55.918082    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbcc5c2-024d-460f-bd94-9ba4aba17eb9-kube-api-access-9km5t\" (UniqueName: \"kubernetes.io/projected/dbbcc5c2-024d-460f-bd94-9ba4aba17eb9-kube-api-access-9km5t\") pod \"dbbcc5c2-024d-460f-bd94-9ba4aba17eb9\" (UID: \"dbbcc5c2-024d-460f-bd94-9ba4aba17eb9\") "
Nov 01 22:53:55.920236 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:55.920207    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbcc5c2-024d-460f-bd94-9ba4aba17eb9-kube-api-access-9km5t" pod "dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" (UID: "dbbcc5c2-024d-460f-bd94-9ba4aba17eb9"). InnerVolumeSpecName "kube-api-access-9km5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:53:56.020258 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.020225    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9km5t\" (UniqueName: \"kubernetes.io/projected/dbbcc5c2-024d-460f-bd94-9ba4aba17eb9-kube-api-access-9km5t\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:56.490258 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.490219    4837 generic.go:356] "Generic (PLEG): container finished" podID="dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" containerID="0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c" exitCode=137
Nov 01 22:53:56.490436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.490272    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:56.490436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.490315    4837 scope.go:122] "RemoveContainer" containerID="0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c"
Nov 01 22:53:56.503006 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.502973    4837 scope.go:122] "RemoveContainer" containerID="0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c"
Nov 01 22:53:56.503537 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:53:56.503406    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c\": container with ID starting with 0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c not found: ID does not exist" containerID="0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c"
Nov 01 22:53:56.503537 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.503440    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c"} err="failed to get container status \"0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c\": rpc error: code = NotFound desc = could not find container \"0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c\": container with ID starting with 0a51c73e13c64e766ca1ef4c5fc8814afd8baedb5b0b465df6c5fd1d8392475c not found: ID does not exist"
Nov 01 22:53:56.506954 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.506930    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"]
Nov 01 22:53:56.507625 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.507610    4837 cpu_manager.go:414] "containerMap: removing container" podUID="dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" containerName="test-curl-7137-2f1e1b"
Nov 01 22:53:56.507706 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.507626    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" containerName="test-curl-7137-2f1e1b"
Nov 01 22:53:56.507764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.507719    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" containerName="test-curl-7137-2f1e1b"
Nov 01 22:53:56.512540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.512521    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.556269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.548398    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" path="/var/lib/kubelet/pods/dbbcc5c2-024d-460f-bd94-9ba4aba17eb9/volumes"
Nov 01 22:53:56.556269 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:53:56.552312    4837 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-2f1e1b\" is forbidden: User \"system:node:ip-10-0-129-77.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-77.us-west-2.compute.internal' and this object" podUID="dbbcc5c2-024d-460f-bd94-9ba4aba17eb9" pod="opendatahub/test-curl-7137-2f1e1b"
Nov 01 22:53:56.602114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.599663    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-661e9f-openshift-default-dockercfg-z95wb\""
Nov 01 22:53:56.615715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.615659    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"]
Nov 01 22:53:56.636530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636494    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-socket\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.636697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636550    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-credential-socket\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.636697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636595    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-data\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.636697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636686    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istiod-ca-cert\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.636905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636709    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-podinfo\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.636905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636756    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-certs\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.636905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636775    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-envoy\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.636905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636811    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-token\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.636905 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.636849    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s49tf\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-kube-api-access-s49tf\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751272    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-socket\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751377    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-credential-socket\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751465    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-data\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751631    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istiod-ca-cert\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751675    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-podinfo\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751774    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-certs\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751807    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-envoy\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751879    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-token\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.769337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.751942    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s49tf\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-kube-api-access-s49tf\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.832079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.787449    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-socket\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.832079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.787725    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-certs\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.832079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.808708    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-podinfo\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.832079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.821224    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-data\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.832079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.822580    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istiod-ca-cert\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:56.832079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:56.828443    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-credential-socket\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:57.145623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:57.144698    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-envoy\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:57.146342 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:57.145558    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-token\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:57.148304 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:57.148285    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s49tf\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-kube-api-access-s49tf\") pod \"e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") " pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:57.203097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:57.200532    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:57.706368 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:53:57.706341    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc510ab0f_4027_4c03_aead_96adc4e1cb3a.slice/crio-897461cebc48117b203ca20238fa51e44af971b75c561667d649c42da9dab164 WatchSource:0}: Error finding container 897461cebc48117b203ca20238fa51e44af971b75c561667d649c42da9dab164: Status 404 returned error can't find the container with id 897461cebc48117b203ca20238fa51e44af971b75c561667d649c42da9dab164
Nov 01 22:53:57.706682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:57.706661    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"]
Nov 01 22:53:57.709298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:57.709265    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:57.709370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:57.709331    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:57.709370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:57.709361    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:53:58.511115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:58.511063    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" event={"ID":"c510ab0f-4027-4c03-aead-96adc4e1cb3a","Type":"ContainerStarted","Data":"31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd"}
Nov 01 22:53:58.511614 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:58.511140    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" event={"ID":"c510ab0f-4027-4c03-aead-96adc4e1cb3a","Type":"ContainerStarted","Data":"897461cebc48117b203ca20238fa51e44af971b75c561667d649c42da9dab164"}
Nov 01 22:53:58.671229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:58.671176    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" podStartSLOduration=2.671162298 podStartE2EDuration="2.671162298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:58.670764908 +0000 UTC m=+1820.259963978" watchObservedRunningTime="2026-11-01 22:53:58.671162298 +0000 UTC m=+1820.260361357"
Nov 01 22:53:59.048267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.048232    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"]
Nov 01 22:53:59.052256 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.052211    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c368b5-8xkbj"]
Nov 01 22:53:59.206739 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.206652    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:59.213324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.213297    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:59.542223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.542192    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:59.543167 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.543142    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:53:59.710622 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.708264    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"]
Nov 01 22:53:59.716189 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.716165    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799302    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f70dbb-d022-4392-b469-c9b518f8592d-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799388 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799344    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799409    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799520    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799604 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799596    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799678    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799704    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799737    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f70dbb-d022-4392-b469-c9b518f8592d-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.799917 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.799818    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65cbl\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-kube-api-access-65cbl\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.927065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.873371    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"]
Nov 01 22:53:59.928029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.927991    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f70dbb-d022-4392-b469-c9b518f8592d-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.928029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.928064    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.928266 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.928096    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.928266 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.928158    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.928266 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.928228    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.928463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.928310    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.928463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.928336    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.928463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.928370    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f70dbb-d022-4392-b469-c9b518f8592d-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.928463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.928454    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65cbl\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-kube-api-access-65cbl\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.965184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.941159    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.965184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.941758    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.965184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.942018    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.965184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.942300    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.965184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.944747    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f70dbb-d022-4392-b469-c9b518f8592d-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.965184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.964487    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.965683 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.965643    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f70dbb-d022-4392-b469-c9b518f8592d-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.982832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.982805    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65cbl\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-kube-api-access-65cbl\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:53:59.993909 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:53:59.993888    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:54:00.035552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:00.035524    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:54:00.421759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:00.421733    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 22:54:00.528911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:00.528856    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250422a9-1111-43d5-9683-85cfa42c9f50" path="/var/lib/kubelet/pods/250422a9-1111-43d5-9683-85cfa42c9f50/volumes"
Nov 01 22:54:00.687565 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:54:00.687540    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f70dbb_d022_4392_b469_c9b518f8592d.slice/crio-f00164ccf2eb5fd2980f10401c160ebdf5236d00f4a8f5ef52de467a07f00c64 WatchSource:0}: Error finding container f00164ccf2eb5fd2980f10401c160ebdf5236d00f4a8f5ef52de467a07f00c64: Status 404 returned error can't find the container with id f00164ccf2eb5fd2980f10401c160ebdf5236d00f4a8f5ef52de467a07f00c64
Nov 01 22:54:00.688582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:00.688557    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"]
Nov 01 22:54:00.691000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:00.690950    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:00.691115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:00.691054    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:00.691115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:00.691103    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:01.573225 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:01.573192    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" event={"ID":"38f70dbb-d022-4392-b469-c9b518f8592d","Type":"ContainerStarted","Data":"aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd"}
Nov 01 22:54:01.573393 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:01.573237    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" event={"ID":"38f70dbb-d022-4392-b469-c9b518f8592d","Type":"ContainerStarted","Data":"f00164ccf2eb5fd2980f10401c160ebdf5236d00f4a8f5ef52de467a07f00c64"}
Nov 01 22:54:01.764506 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:01.764432    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podStartSLOduration=2.764409877 podStartE2EDuration="2.764409877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:01.763032375 +0000 UTC m=+1823.352231444" watchObservedRunningTime="2026-11-01 22:54:01.764409877 +0000 UTC m=+1823.353608937"
Nov 01 22:54:01.964294 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:01.964262    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:54:01.964489 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:01.964334    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:54:02.040090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.036320    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:54:02.055125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.049165    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 22:54:02.055125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.049244    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 22:54:02.625068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.624221    4837 generic.go:356] "Generic (PLEG): container finished" podID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerID="3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6" exitCode=0
Nov 01 22:54:02.626121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.625515    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" event={"ID":"70d77d32-49a4-405f-af93-db6b4fc0e303","Type":"ContainerDied","Data":"3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6"}
Nov 01 22:54:02.626121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.625979    4837 scope.go:122] "RemoveContainer" containerID="3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6"
Nov 01 22:54:02.634491 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.634453    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:02.634749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.634724    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:02.634962 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:02.634940    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:03.053066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:03.037400    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 22:54:03.053066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:03.037465    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 22:54:03.655067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:03.654078    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" event={"ID":"70d77d32-49a4-405f-af93-db6b4fc0e303","Type":"ContainerStarted","Data":"3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3"}
Nov 01 22:54:04.040062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:04.038532    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 22:54:04.040062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:04.038608    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 22:54:04.087755 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:04.087536    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:54:04.087755 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:04.087646    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:54:04.087755 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:04.087686    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:54:04.216095 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:04.216057    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"]
Nov 01 22:54:04.216318 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:04.216292    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy" containerID="cri-o://a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f" gracePeriod=30
Nov 01 22:54:05.037421 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:05.037384    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 22:54:05.037711 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:05.037681    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 22:54:05.082179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:05.081474    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:54:05.082179 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:05.081530    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:54:05.695065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:05.694830    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:54:06.167120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.158648    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 22:54:06.167120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.158716    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 22:54:06.226070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.221576    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:54:06.318067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.313518    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:54:06.513115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.506441    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"]
Nov 01 22:54:06.686068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.678374    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"]
Nov 01 22:54:06.724088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.723622    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:06.833390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.821361    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-661e9f\""
Nov 01 22:54:06.833390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.821471    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-661e9f-dockercfg-msvbs\""
Nov 01 22:54:06.905285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.895777    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b26295-9a04-41fb-8c2d-7633791fd7f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq\" (UID: \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:06.905285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:06.895957    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h298\" (UniqueName: \"kubernetes.io/projected/e5b26295-9a04-41fb-8c2d-7633791fd7f9-kube-api-access-2h298\") pod \"payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq\" (UID: \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:07.029069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.003551    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b26295-9a04-41fb-8c2d-7633791fd7f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq\" (UID: \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:07.029069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.003696    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h298\" (UniqueName: \"kubernetes.io/projected/e5b26295-9a04-41fb-8c2d-7633791fd7f9-kube-api-access-2h298\") pod \"payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq\" (UID: \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:07.030143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.030110    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b26295-9a04-41fb-8c2d-7633791fd7f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq\" (UID: \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:07.043254 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.043217    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 22:54:07.043534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.043488    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 22:54:07.096310 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.096225    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:54:07.096555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.096527    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:54:07.123182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.121056    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h298\" (UniqueName: \"kubernetes.io/projected/e5b26295-9a04-41fb-8c2d-7633791fd7f9-kube-api-access-2h298\") pod \"payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq\" (UID: \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:07.163359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.162006    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:07.232543 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.231914    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:07.312485 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.312168    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:07.334681 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.333174    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:07.396540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.396251    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:07.425066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.413273    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:54:07.518070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.514542    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:07.539636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.539580    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:07.585112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.582406    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 01 22:54:07.659220 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:07.636905    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:54:08.059356 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.057966    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"]
Nov 01 22:54:08.060289 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.059830    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 22:54:08.060289 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.059896    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 22:54:08.080967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.071617    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:54:08.080967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.071674    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:54:08.527591 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.527556    4837 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:54:08.527755 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.527611    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:54:08.824268 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.824229    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq" event={"ID":"e5b26295-9a04-41fb-8c2d-7633791fd7f9","Type":"ContainerStarted","Data":"ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240"}
Nov 01 22:54:08.824459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.824298    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq" event={"ID":"e5b26295-9a04-41fb-8c2d-7633791fd7f9","Type":"ContainerStarted","Data":"e9e2a23f8db1a207b3e85f78de2a7c086a22293cf17d9064813c6bdb832e7030"}
Nov 01 22:54:08.824459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.824445    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:08.960206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:08.960151    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq" podStartSLOduration=2.960135434 podStartE2EDuration="2.960135434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:08.955516964 +0000 UTC m=+1830.544716025" watchObservedRunningTime="2026-11-01 22:54:08.960135434 +0000 UTC m=+1830.549334553"
Nov 01 22:54:09.059024 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.058994    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:54:09.059339 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.059318    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:54:09.060034 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.060014    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 22:54:09.072128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.071653    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:54:09.072128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.071709    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:54:09.625654 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.625631    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:54:09.784651 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.784624    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-envoy\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.784825 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.784665    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-token\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-token\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.784895 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.784848    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-data\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.784940 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.784893    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-podinfo\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.784993 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.784977    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-kube-api-access-l2chs\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-kube-api-access-l2chs\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.785088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785010    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-credential-socket\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.785166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785099    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-socket\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.785166 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785149    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0562f9bc-b7b9-4885-9050-35e6200c19a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562f9bc-b7b9-4885-9050-35e6200c19a7-istiod-ca-cert\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.785274 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785186    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-data" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.785337 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785286    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-certs\") pod \"0562f9bc-b7b9-4885-9050-35e6200c19a7\" (UID: \"0562f9bc-b7b9-4885-9050-35e6200c19a7\") "
Nov 01 22:54:09.785415 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785390    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-socket" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.785716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785582    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0562f9bc-b7b9-4885-9050-35e6200c19a7-istiod-ca-cert" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:09.785716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785587    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-credential-socket" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.785716 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.785670    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-certs" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.786130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.786113    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.786130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.786131    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.786253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.786140    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.786253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.786148    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0562f9bc-b7b9-4885-9050-35e6200c19a7-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.786253 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.786157    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.787623 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.787597    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-token" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:09.788259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.788240    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-podinfo" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:54:09.788693 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.788674    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-envoy" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.789913 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.789893    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-kube-api-access-l2chs" pod "0562f9bc-b7b9-4885-9050-35e6200c19a7" (UID: "0562f9bc-b7b9-4885-9050-35e6200c19a7"). InnerVolumeSpecName "kube-api-access-l2chs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:09.858029 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.857942    4837 generic.go:356] "Generic (PLEG): container finished" podID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerID="a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f" exitCode=0
Nov 01 22:54:09.858304 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.858281    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" event={"ID":"0562f9bc-b7b9-4885-9050-35e6200c19a7","Type":"ContainerDied","Data":"a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f"}
Nov 01 22:54:09.858387 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.858371    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f" event={"ID":"0562f9bc-b7b9-4885-9050-35e6200c19a7","Type":"ContainerDied","Data":"701595d09e98d44ca8a9818c147f2666490153776d822a0d0bbf260c9f409cac"}
Nov 01 22:54:09.858460 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.858434    4837 scope.go:122] "RemoveContainer" containerID="a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f"
Nov 01 22:54:09.858621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.858605    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"
Nov 01 22:54:09.887608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.887584    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2chs\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-kube-api-access-l2chs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.887697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.887613    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.887697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.887622    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.887697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.887630    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0562f9bc-b7b9-4885-9050-35e6200c19a7-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.887697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.887669    4837 scope.go:122] "RemoveContainer" containerID="f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a"
Nov 01 22:54:09.897074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.897016    4837 scope.go:122] "RemoveContainer" containerID="a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f"
Nov 01 22:54:09.897365 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:09.897340    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f\": container with ID starting with a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f not found: ID does not exist" containerID="a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f"
Nov 01 22:54:09.897462 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.897371    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f"} err="failed to get container status \"a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f\": rpc error: code = NotFound desc = could not find container \"a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f\": container with ID starting with a607728ebdd75b6c559c7a439356d4a3e1833ec8f76ddf2422635e01c3c3195f not found: ID does not exist"
Nov 01 22:54:09.897462 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.897389    4837 scope.go:122] "RemoveContainer" containerID="f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a"
Nov 01 22:54:09.897676 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:09.897659    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a\": container with ID starting with f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a not found: ID does not exist" containerID="f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a"
Nov 01 22:54:09.897730 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:09.897681    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a"} err="failed to get container status \"f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a\": rpc error: code = NotFound desc = could not find container \"f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a\": container with ID starting with f573dd56ef9e229b63780c062390b534d484f460451c61f9a740476f90890f6a not found: ID does not exist"
Nov 01 22:54:10.018061 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:10.018015    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"]
Nov 01 22:54:10.021350 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:10.021331    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-sld8f"]
Nov 01 22:54:10.071403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:10.071369    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:54:10.071772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:10.071419    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:54:10.160632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:10.160560    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"]
Nov 01 22:54:10.537606 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:10.537338    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" path="/var/lib/kubelet/pods/0562f9bc-b7b9-4885-9050-35e6200c19a7/volumes"
Nov 01 22:54:11.075188 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:11.075157    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:54:11.075659 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:11.075607    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:54:11.076327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:11.076298    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 22:54:15.161438 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:15.161398    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579" podUID="fa512302-8ce9-4aec-95ce-1249b284107e" containerName="payload-processing" containerID="cri-o://9f2fd08824e9aed4ea8eb7c705fed96ff7652d6b7c3a00033dc0bb492309610a" gracePeriod=25
Nov 01 22:54:16.039925 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:16.039893    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"]
Nov 01 22:54:16.040169 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:16.040140    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" containerID="cri-o://7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4" gracePeriod=30
Nov 01 22:54:16.181819 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:16.181770    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"]
Nov 01 22:54:16.182431 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:16.182030    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" containerID="cri-o://c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018" gracePeriod=30
Nov 01 22:54:16.202701 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:16.202675    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"]
Nov 01 22:54:16.202886 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:16.202866    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy" containerID="cri-o://f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe" gracePeriod=30
Nov 01 22:54:18.294542 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:18.294469    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:54:18.294542 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:18.294527    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:54:18.432560 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:18.432523    4837 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:54:18.432707 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:18.432593    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:54:18.819632 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:18.819568    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:54:19.863102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:19.863071    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 22:54:21.358595 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.358323    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:54:21.439832 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.439795    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-certs\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.440182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.439953    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-envoy\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.440182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.439995    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-istio-token\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-istio-token\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.440182 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.440026    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/583275c8-2239-4b6e-9236-307cea25f813-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583275c8-2239-4b6e-9236-307cea25f813-istio-podinfo\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.440619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.440189    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-data\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.440619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.440260    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-credential-socket\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.440619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.440289    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-socket\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.440619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.440354    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583275c8-2239-4b6e-9236-307cea25f813-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583275c8-2239-4b6e-9236-307cea25f813-istiod-ca-cert\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.440619 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.440489    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-kube-api-access-nl6gf\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-kube-api-access-nl6gf\") pod \"583275c8-2239-4b6e-9236-307cea25f813\" (UID: \"583275c8-2239-4b6e-9236-307cea25f813\") "
Nov 01 22:54:21.443443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.442660    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-certs" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.443443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.442840    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-socket" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.443443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.442996    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-data" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.445081 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.445017    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-credential-socket" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.445193 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.445082    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583275c8-2239-4b6e-9236-307cea25f813-istiod-ca-cert" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:21.447471 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.447421    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579" podUID="fa512302-8ce9-4aec-95ce-1249b284107e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:54:21.448071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.448030    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-envoy" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.453513 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.453488    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/583275c8-2239-4b6e-9236-307cea25f813-istio-podinfo" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:54:21.453613 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.453540    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-kube-api-access-nl6gf" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "kube-api-access-nl6gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.455452 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.455424    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-istio-token" pod "583275c8-2239-4b6e-9236-307cea25f813" (UID: "583275c8-2239-4b6e-9236-307cea25f813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.474368 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.474346    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:54:21.512567 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.512539    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:54:21.543930 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.543831    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-envoy\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.544247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.544214    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-podinfo\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.545790 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.545756    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4947b6-04b4-45a2-b98b-19d92bd16520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4947b6-04b4-45a2-b98b-19d92bd16520-istiod-ca-cert\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.545918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.545841    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-data\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.545918 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.545881    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-socket\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.546117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.545942    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-token\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-token\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.546117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.546060    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-kube-api-access-656xs\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-kube-api-access-656xs\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.546117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.546109    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-certs\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.546290 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.546181    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-credential-socket\") pod \"db4947b6-04b4-45a2-b98b-19d92bd16520\" (UID: \"db4947b6-04b4-45a2-b98b-19d92bd16520\") "
Nov 01 22:54:21.546290 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.546245    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-socket" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.546412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.546395    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-data" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.546768 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.546750    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4947b6-04b4-45a2-b98b-19d92bd16520-istiod-ca-cert" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:21.547125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547103    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4947b6-04b4-45a2-b98b-19d92bd16520-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547128    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547148    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547168    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-envoy" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.547237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547194    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547213    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547226    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547245    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583275c8-2239-4b6e-9236-307cea25f813-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547258    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547270    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547288    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583275c8-2239-4b6e-9236-307cea25f813-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547301    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583275c8-2239-4b6e-9236-307cea25f813-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547315    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl6gf\" (UniqueName: \"kubernetes.io/projected/583275c8-2239-4b6e-9236-307cea25f813-kube-api-access-nl6gf\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.547555 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547507    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-certs" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.547828 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.547744    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-credential-socket" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.548714 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.548683    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-podinfo" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:54:21.552309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.552287    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-kube-api-access-656xs" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "kube-api-access-656xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.555385 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.555363    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-token" pod "db4947b6-04b4-45a2-b98b-19d92bd16520" (UID: "db4947b6-04b4-45a2-b98b-19d92bd16520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.648927 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.648811    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-socket\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.648927 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.648903    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-envoy\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.649197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.648965    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-credential-socket\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.649197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649090    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c7401d-383e-4522-8b28-68f5bf2413fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c7401d-383e-4522-8b28-68f5bf2413fc-istiod-ca-cert\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.649197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649138    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-kube-api-access-4bzmx\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-kube-api-access-4bzmx\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.649197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649192    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-data\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.649423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649200    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-socket" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.649423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649280    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-certs\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.649423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649384    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-token\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-token\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.649598 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649512    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-podinfo\") pod \"15c7401d-383e-4522-8b28-68f5bf2413fc\" (UID: \"15c7401d-383e-4522-8b28-68f5bf2413fc\") "
Nov 01 22:54:21.649598 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649568    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c7401d-383e-4522-8b28-68f5bf2413fc-istiod-ca-cert" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:21.650394 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.649821    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-credential-socket" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.650394 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650083    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-certs" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.650394 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650380    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650394 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650391    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-data" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650402    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650419    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650432    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650445    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c7401d-383e-4522-8b28-68f5bf2413fc-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650456    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650464    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650472    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-656xs\" (UniqueName: \"kubernetes.io/projected/db4947b6-04b4-45a2-b98b-19d92bd16520-kube-api-access-656xs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650481    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.650671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.650490    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4947b6-04b4-45a2-b98b-19d92bd16520-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.651792 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.651768    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-kube-api-access-4bzmx" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "kube-api-access-4bzmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.652287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.652264    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-token" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.652287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.652275    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-podinfo" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:54:21.655589 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.655569    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-envoy" pod "15c7401d-383e-4522-8b28-68f5bf2413fc" (UID: "15c7401d-383e-4522-8b28-68f5bf2413fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.752444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.752416    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.752444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.752441    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.752444 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.752451    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.752653 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.752461    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzmx\" (UniqueName: \"kubernetes.io/projected/15c7401d-383e-4522-8b28-68f5bf2413fc-kube-api-access-4bzmx\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.752653 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:21.752470    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c7401d-383e-4522-8b28-68f5bf2413fc-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:22.047672 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.047637    4837 generic.go:356] "Generic (PLEG): container finished" podID="583275c8-2239-4b6e-9236-307cea25f813" containerID="7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4" exitCode=0
Nov 01 22:54:22.047869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.047698    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"
Nov 01 22:54:22.047869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.047715    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" event={"ID":"583275c8-2239-4b6e-9236-307cea25f813","Type":"ContainerDied","Data":"7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4"}
Nov 01 22:54:22.047869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.047807    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx" event={"ID":"583275c8-2239-4b6e-9236-307cea25f813","Type":"ContainerDied","Data":"a26ae1ded572860b9c79a8235c8eeea8e3ec244a13eb83af5b69c929d32bc7e7"}
Nov 01 22:54:22.047869 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.047837    4837 scope.go:122] "RemoveContainer" containerID="7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4"
Nov 01 22:54:22.049340 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.049312    4837 generic.go:356] "Generic (PLEG): container finished" podID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerID="f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe" exitCode=0
Nov 01 22:54:22.049463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.049367    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"
Nov 01 22:54:22.049463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.049390    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" event={"ID":"15c7401d-383e-4522-8b28-68f5bf2413fc","Type":"ContainerDied","Data":"f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe"}
Nov 01 22:54:22.049463 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.049438    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r" event={"ID":"15c7401d-383e-4522-8b28-68f5bf2413fc","Type":"ContainerDied","Data":"dc02bccd3f2aab6a5d2123ed5c9099b6f9bb07e50a6555a8df0495e0a990a138"}
Nov 01 22:54:22.050977 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.050952    4837 generic.go:356] "Generic (PLEG): container finished" podID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerID="c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018" exitCode=0
Nov 01 22:54:22.051099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.051034    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" event={"ID":"db4947b6-04b4-45a2-b98b-19d92bd16520","Type":"ContainerDied","Data":"c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018"}
Nov 01 22:54:22.051099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.051088    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"
Nov 01 22:54:22.051220 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.051099    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57" event={"ID":"db4947b6-04b4-45a2-b98b-19d92bd16520","Type":"ContainerDied","Data":"6fa517c021e10624d52669f9e190ba98bb9b15ac437a5a648bed124b5adda629"}
Nov 01 22:54:22.064540 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.064509    4837 scope.go:122] "RemoveContainer" containerID="7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4"
Nov 01 22:54:22.064821 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:22.064801    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4\": container with ID starting with 7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4 not found: ID does not exist" containerID="7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4"
Nov 01 22:54:22.064928 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.064828    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4"} err="failed to get container status \"7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4\": rpc error: code = NotFound desc = could not find container \"7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4\": container with ID starting with 7618a4e66b73c41bf288f22aa6dfa36669ac7ed6a9298cab417aad23015e47d4 not found: ID does not exist"
Nov 01 22:54:22.064928 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.064851    4837 scope.go:122] "RemoveContainer" containerID="f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe"
Nov 01 22:54:22.092523 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.092508    4837 scope.go:122] "RemoveContainer" containerID="f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe"
Nov 01 22:54:22.092803 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:22.092786    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe\": container with ID starting with f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe not found: ID does not exist" containerID="f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe"
Nov 01 22:54:22.092909 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.092814    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe"} err="failed to get container status \"f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe\": rpc error: code = NotFound desc = could not find container \"f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe\": container with ID starting with f4f4a5d1336a00669989bf61ca8ed2d9cd50434730d8c099b3d652b3c975aabe not found: ID does not exist"
Nov 01 22:54:22.092909 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.092843    4837 scope.go:122] "RemoveContainer" containerID="c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018"
Nov 01 22:54:22.102130 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.102109    4837 scope.go:122] "RemoveContainer" containerID="c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018"
Nov 01 22:54:22.102399 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:22.102378    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018\": container with ID starting with c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018 not found: ID does not exist" containerID="c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018"
Nov 01 22:54:22.102477 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.102406    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018"} err="failed to get container status \"c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018\": rpc error: code = NotFound desc = could not find container \"c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018\": container with ID starting with c4b75755847a075daeec0d00407e8c43986c6a916023ed7c15f2effeefb42018 not found: ID does not exist"
Nov 01 22:54:22.204354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.204323    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"]
Nov 01 22:54:22.250693 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.250668    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-sqjkx"]
Nov 01 22:54:22.321192 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.321099    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"]
Nov 01 22:54:22.325483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.325455    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-7gb57"]
Nov 01 22:54:22.432517 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.432486    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"]
Nov 01 22:54:22.435774 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.435752    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-fkh6r"]
Nov 01 22:54:22.527416 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.527386    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" path="/var/lib/kubelet/pods/15c7401d-383e-4522-8b28-68f5bf2413fc/volumes"
Nov 01 22:54:22.527955 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.527934    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583275c8-2239-4b6e-9236-307cea25f813" path="/var/lib/kubelet/pods/583275c8-2239-4b6e-9236-307cea25f813/volumes"
Nov 01 22:54:22.528476 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:22.528457    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" path="/var/lib/kubelet/pods/db4947b6-04b4-45a2-b98b-19d92bd16520/volumes"
Nov 01 22:54:31.436959 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:31.436921    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579" podUID="fa512302-8ce9-4aec-95ce-1249b284107e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:54:36.279067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.246152    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:36.355582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.274270    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:36.394610 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.274341    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:36.421868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.274379    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:54:36.421868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.417977    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:36.421868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.419461    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:36.421868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.420409    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:54:36.529932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.529832    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:54:36.551076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.549236    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:36.575790 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.575126    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:36.681067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.680856    4837 generic.go:356] "Generic (PLEG): container finished" podID="fa512302-8ce9-4aec-95ce-1249b284107e" containerID="9f2fd08824e9aed4ea8eb7c705fed96ff7652d6b7c3a00033dc0bb492309610a" exitCode=0
Nov 01 22:54:36.744067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.743812    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579" event={"ID":"fa512302-8ce9-4aec-95ce-1249b284107e","Type":"ContainerDied","Data":"9f2fd08824e9aed4ea8eb7c705fed96ff7652d6b7c3a00033dc0bb492309610a"}
Nov 01 22:54:36.753104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.750530    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:54:36.814131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.813614    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa512302-8ce9-4aec-95ce-1249b284107e-kube-api-access-mk9p2\" (UniqueName: \"kubernetes.io/projected/fa512302-8ce9-4aec-95ce-1249b284107e-kube-api-access-mk9p2\") pod \"fa512302-8ce9-4aec-95ce-1249b284107e\" (UID: \"fa512302-8ce9-4aec-95ce-1249b284107e\") "
Nov 01 22:54:36.814131 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.813719    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa512302-8ce9-4aec-95ce-1249b284107e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa512302-8ce9-4aec-95ce-1249b284107e-plugins-config-volume\") pod \"fa512302-8ce9-4aec-95ce-1249b284107e\" (UID: \"fa512302-8ce9-4aec-95ce-1249b284107e\") "
Nov 01 22:54:36.826065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.816360    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa512302-8ce9-4aec-95ce-1249b284107e-plugins-config-volume" pod "fa512302-8ce9-4aec-95ce-1249b284107e" (UID: "fa512302-8ce9-4aec-95ce-1249b284107e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:36.887396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.883099    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa512302-8ce9-4aec-95ce-1249b284107e-kube-api-access-mk9p2" pod "fa512302-8ce9-4aec-95ce-1249b284107e" (UID: "fa512302-8ce9-4aec-95ce-1249b284107e"). InnerVolumeSpecName "kube-api-access-mk9p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:36.945067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.934033    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa512302-8ce9-4aec-95ce-1249b284107e-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:36.945067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:36.934107    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk9p2\" (UniqueName: \"kubernetes.io/projected/fa512302-8ce9-4aec-95ce-1249b284107e-kube-api-access-mk9p2\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:37.359886 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.359831    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:37.361724 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.361702    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:54:37.362547 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.359772    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:37.366323 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.365270    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:54:37.421152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.421110    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:37.422355 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.421969    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:37.431062 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.421999    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Nov 01 22:54:37.440121 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.422560    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:54:37.440321 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.440306    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:54:37.464515 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.447127    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:37.464976 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.464950    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:54:37.577851 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.577810    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:37.585325 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.585298    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:54:37.786974 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.786945    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 22:54:37.787353 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.787339    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 22:54:37.805632 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.805602    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:54:37.811391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.811370    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:54:37.877070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.861433    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:37.877070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.870177    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"
Nov 01 22:54:37.891787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.886002    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579" event={"ID":"fa512302-8ce9-4aec-95ce-1249b284107e","Type":"ContainerDied","Data":"556c0e3f46e9b8e801a687b6db1e9a25b6d36a6525ed535dbf763bc819334fc8"}
Nov 01 22:54:37.891787 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.886452    4837 scope.go:122] "RemoveContainer" containerID="9f2fd08824e9aed4ea8eb7c705fed96ff7652d6b7c3a00033dc0bb492309610a"
Nov 01 22:54:37.913000 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.911515    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:54:37.914203 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:37.914179    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:54:38.685239 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:38.685206    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"]
Nov 01 22:54:38.691699 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:38.691669    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-qs579"]
Nov 01 22:54:40.527971 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:40.527936    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa512302-8ce9-4aec-95ce-1249b284107e" path="/var/lib/kubelet/pods/fa512302-8ce9-4aec-95ce-1249b284107e/volumes"
Nov 01 22:54:42.305180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:42.305145    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:54:42.305552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:42.305197    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:54:43.921279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:43.921236    4837 generic.go:356] "Generic (PLEG): container finished" podID="80e21ad7-e679-4381-a949-814c8b5207cd" containerID="ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f" exitCode=0
Nov 01 22:54:43.922197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:43.921317    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" event={"ID":"80e21ad7-e679-4381-a949-814c8b5207cd","Type":"ContainerDied","Data":"ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f"}
Nov 01 22:54:43.922197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:43.921739    4837 scope.go:122] "RemoveContainer" containerID="ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f"
Nov 01 22:54:43.924231 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:43.924200    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:43.924344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:43.924266    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:43.924344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:43.924296    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:44.952082 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:44.936196    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" event={"ID":"80e21ad7-e679-4381-a949-814c8b5207cd","Type":"ContainerStarted","Data":"80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8"}
Nov 01 22:54:44.955761 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:44.955726    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:54:45.097107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097072    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"]
Nov 01 22:54:45.097764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097747    4837 cpu_manager.go:414] "containerMap: removing container" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy"
Nov 01 22:54:45.097764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097766    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097781    4837 cpu_manager.go:414] "containerMap: removing container" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097786    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097822    4837 cpu_manager.go:414] "containerMap: removing container" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097828    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097844    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097850    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097857    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097861    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097868    4837 cpu_manager.go:414] "containerMap: removing container" podUID="fa512302-8ce9-4aec-95ce-1249b284107e" containerName="payload-processing"
Nov 01 22:54:45.097923 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097872    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa512302-8ce9-4aec-95ce-1249b284107e" containerName="payload-processing"
Nov 01 22:54:45.098264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097965    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy"
Nov 01 22:54:45.098264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097972    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0562f9bc-b7b9-4885-9050-35e6200c19a7" containerName="istio-proxy"
Nov 01 22:54:45.098264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097979    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa512302-8ce9-4aec-95ce-1249b284107e" containerName="payload-processing"
Nov 01 22:54:45.098264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097985    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4947b6-04b4-45a2-b98b-19d92bd16520" containerName="istio-proxy"
Nov 01 22:54:45.098264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.097993    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="583275c8-2239-4b6e-9236-307cea25f813" containerName="istio-proxy"
Nov 01 22:54:45.098264 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.098004    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c7401d-383e-4522-8b28-68f5bf2413fc" containerName="istio-proxy"
Nov 01 22:54:45.101629 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.101600    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.265327 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.265235    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.265650 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.265624    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.265807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.265787    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.265966 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.265949    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.266116 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.265976    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.266116 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.266066    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.266116 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.266105    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.266293 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.266270    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.266366 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.266345    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzlvk\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-kube-api-access-qzlvk\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.268183 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.268154    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"]
Nov 01 22:54:45.368351 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368308    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368381    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368478    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368541 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368503    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368706 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368582    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368706 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368626    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368789    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368810 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368793    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368942 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368904    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzlvk\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-kube-api-access-qzlvk\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.368999 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.368967    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.369273 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.369215    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.369588 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.369559    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.369695 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.369674    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.369997 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.369961    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.372076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.371859    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.372076 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.371879    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.425135 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.425113    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.425311 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.425290    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzlvk\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-kube-api-access-qzlvk\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.715262 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.715234    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:45.957109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.956381    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:54:45.957109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:45.956437    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:54:46.163242 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:46.163214    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"]
Nov 01 22:54:46.164614 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:54:46.164586    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ae3b0c_b48c_4fcc_83ef_0e18aad2fe95.slice/crio-c305ce0f76c12df418dea2c25a6ac78414f3ef9d9455d9cce9787472d1f16415 WatchSource:0}: Error finding container c305ce0f76c12df418dea2c25a6ac78414f3ef9d9455d9cce9787472d1f16415: Status 404 returned error can't find the container with id c305ce0f76c12df418dea2c25a6ac78414f3ef9d9455d9cce9787472d1f16415
Nov 01 22:54:46.167381 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:46.167349    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:46.167488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:46.167423    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:46.167488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:46.167470    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:46.984069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:46.968449    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:54:46.984069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:46.968539    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:54:46.998215 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:46.998177    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" event={"ID":"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95","Type":"ContainerStarted","Data":"e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995"}
Nov 01 22:54:46.998405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:46.998249    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" event={"ID":"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95","Type":"ContainerStarted","Data":"c305ce0f76c12df418dea2c25a6ac78414f3ef9d9455d9cce9787472d1f16415"}
Nov 01 22:54:47.152065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:47.141322    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" podStartSLOduration=2.14130347 podStartE2EDuration="2.14130347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:47.139169585 +0000 UTC m=+1868.728368648" watchObservedRunningTime="2026-11-01 22:54:47.14130347 +0000 UTC m=+1868.730502531"
Nov 01 22:54:47.717066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:47.716925    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:47.729068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:47.719377    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:54:47.729068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:47.719440    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:54:47.965069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:47.961495    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:54:47.965069 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:47.961581    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:54:48.716542 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:48.716507    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:54:48.716946 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:48.716554    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:54:48.821304 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:48.821269    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:54:48.956446 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:48.956408    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:54:48.956725 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:48.956697    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:54:49.715862 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:49.715827    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:54:49.716035 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:49.715875    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:54:49.958067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:49.957390    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:54:49.958067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:49.957447    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:54:50.716098 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:50.716068    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:54:50.716275 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:50.716118    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:54:50.960425 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:50.960394    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:54:50.960854 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:50.960784    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:54:50.961368 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:50.961352    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 22:54:51.719065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:51.718819    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:54:51.719065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:51.718891    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:54:52.720796 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:52.720767    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:52.721343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:52.721319    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:52.722168 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:52.722151    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 22:54:57.110658 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.110622    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"]
Nov 01 22:54:57.163065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.161238    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.258066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.255199    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-584561-openshift-default-dockercfg-qjzlf\""
Nov 01 22:54:57.278248 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.278215    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-envoy\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.278495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.278472    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-credential-socket\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.278641 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.278628    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-certs\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.278754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.278741    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-podinfo\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.279484 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.279462    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-data\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.301075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.279735    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-socket\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.301417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.301386    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istiod-ca-cert\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.301724 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.301700    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jvk\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-kube-api-access-l6jvk\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.302594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.302573    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-token\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.381813 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.381726    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"]
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408206    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-envoy\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408265    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-credential-socket\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408329    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-certs\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408361    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-podinfo\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408397    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-data\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408527    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-socket\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408591    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istiod-ca-cert\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408670    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jvk\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-kube-api-access-l6jvk\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.416259 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.408737    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-token\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.418954 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.418903    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-data\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.419966 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.419481    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-credential-socket\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.420432 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.420414    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-certs\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.421567 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.421284    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-socket\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.421715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.421697    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-envoy\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.422880 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.422867    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istiod-ca-cert\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.517392 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.517359    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-podinfo\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.521580 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.521553    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jvk\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-kube-api-access-l6jvk\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.521901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.521879    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-token\") pod \"e2e-api-b-584561-openshift-default-95c6887c-vdglb\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") " pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.768180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.768147    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:57.821777 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.821744    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"]
Nov 01 22:54:57.828356 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.828332    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:57.877661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.877372    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-8cc47a-serving-cert\""
Nov 01 22:54:57.888578 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.888542    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"]
Nov 01 22:54:57.917733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.917698    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ebabbb-df4d-496d-95bf-0a6e07b92520-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:57.917733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.917746    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:57.918144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.917911    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhsr\" (UniqueName: \"kubernetes.io/projected/28ebabbb-df4d-496d-95bf-0a6e07b92520-kube-api-access-bxhsr\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:57.918144 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:57.917983    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.019676 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.019607    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhsr\" (UniqueName: \"kubernetes.io/projected/28ebabbb-df4d-496d-95bf-0a6e07b92520-kube-api-access-bxhsr\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.019811 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.019692    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.019883 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.019848    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ebabbb-df4d-496d-95bf-0a6e07b92520-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.019883 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.019868    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.019999 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:58.019848    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-serving-cert: secret "maas-api-e2e-shared-b-w2-8cc47a-serving-cert" not found
Nov 01 22:54:58.019999 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:54:58.019964    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls podName:28ebabbb-df4d-496d-95bf-0a6e07b92520 nodeName:}" failed. No retries permitted until 2026-11-01 22:54:58.519947991 +0000 UTC m=+1880.109147042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls") pod "maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d" (UID: "28ebabbb-df4d-496d-95bf-0a6e07b92520") : secret "maas-api-e2e-shared-b-w2-8cc47a-serving-cert" not found
Nov 01 22:54:58.023785 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.020918    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ebabbb-df4d-496d-95bf-0a6e07b92520-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.038890 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.038867    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.073447 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.073419    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhsr\" (UniqueName: \"kubernetes.io/projected/28ebabbb-df4d-496d-95bf-0a6e07b92520-kube-api-access-bxhsr\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.221389 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.216456    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"]
Nov 01 22:54:58.232908 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.232851    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:58.233107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.232947    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:58.233107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.232982    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:54:58.536023 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.535944    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.538582 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.538543    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:58.764287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:58.764257    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:54:59.144269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:59.144188    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" event={"ID":"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4","Type":"ContainerStarted","Data":"3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846"}
Nov 01 22:54:59.144269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:59.144243    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" event={"ID":"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4","Type":"ContainerStarted","Data":"6caef8b3cac5352f6abecf23d257a33dcdf2ebfd2e935ff63ec90f9e43f54425"}
Nov 01 22:54:59.184195 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:59.184167    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"]
Nov 01 22:54:59.184330 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:54:59.184164    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ebabbb_df4d_496d_95bf_0a6e07b92520.slice/crio-d49a2e50e3108c0e6e8b65d930ccb3bbe18a9c7ab6dd5a8b5d29c71a05d63ee1 WatchSource:0}: Error finding container d49a2e50e3108c0e6e8b65d930ccb3bbe18a9c7ab6dd5a8b5d29c71a05d63ee1: Status 404 returned error can't find the container with id d49a2e50e3108c0e6e8b65d930ccb3bbe18a9c7ab6dd5a8b5d29c71a05d63ee1
Nov 01 22:54:59.295379 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:59.295332    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podStartSLOduration=2.295317904 podStartE2EDuration="2.295317904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:59.291239534 +0000 UTC m=+1880.880438594" watchObservedRunningTime="2026-11-01 22:54:59.295317904 +0000 UTC m=+1880.884516963"
Nov 01 22:54:59.768855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:59.768820    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:54:59.774812 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:54:59.774788    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:55:00.155934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:00.155906    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d" event={"ID":"28ebabbb-df4d-496d-95bf-0a6e07b92520","Type":"ContainerStarted","Data":"d49a2e50e3108c0e6e8b65d930ccb3bbe18a9c7ab6dd5a8b5d29c71a05d63ee1"}
Nov 01 22:55:00.156162 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:00.156142    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:55:00.157097 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:00.157079    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:55:01.238068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:01.222394    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d" event={"ID":"28ebabbb-df4d-496d-95bf-0a6e07b92520","Type":"ContainerStarted","Data":"480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092"}
Nov 01 22:55:01.238068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:01.226133    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:55:01.342023 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:01.341964    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d" podStartSLOduration=3.378741804 podStartE2EDuration="4.341950808s" totalImagesPullingTime="963.209004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:01.338612916 +0000 UTC m=+1882.927811975" watchObservedRunningTime="2026-11-01 22:55:01.341950808 +0000 UTC m=+1882.931149869"
Nov 01 22:55:03.889612 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:03.889577    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"]
Nov 01 22:55:03.894450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:03.894426    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:03.952128 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:03.949472    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f0206-openshift-default-dockercfg-nsm5p\""
Nov 01 22:55:04.005903 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.005786    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"]
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.071733    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istiod-ca-cert\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.071863    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-credential-socket\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.071922    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-envoy\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.071971    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-socket\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.072021    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-token\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.085176    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-certs\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.085282    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-data\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.085351    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swk49\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-kube-api-access-swk49\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.091067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.085575    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-podinfo\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.213807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.213762    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-credential-socket\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214005 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.213850    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-envoy\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214005 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.213887    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-socket\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214005 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.213940    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-token\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.214016    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-certs\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.214111    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-data\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214200 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.214172    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swk49\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-kube-api-access-swk49\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.214295    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-podinfo\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214410 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.214390    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istiod-ca-cert\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.214605 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.214571    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-socket\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.215074 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.215018    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-credential-socket\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.215219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.215194    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-data\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.215476 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.215455    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-certs\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.216642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.216616    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istiod-ca-cert\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.218955 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.218928    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-envoy\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.219451 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.219429    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-podinfo\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.268804 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.268782    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-token\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.268916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.268885    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swk49\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-kube-api-access-swk49\") pod \"e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") " pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.508988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.508897    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:04.936901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.936868    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"]
Nov 01 22:55:04.970524 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:55:04.970491    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8603b6a4_30a6_4b97_b2b9_2d0f80c74365.slice/crio-989091eb8d703b1a7e08b9cdda2332b326d2b6595577192099badff3c5b840f3 WatchSource:0}: Error finding container 989091eb8d703b1a7e08b9cdda2332b326d2b6595577192099badff3c5b840f3: Status 404 returned error can't find the container with id 989091eb8d703b1a7e08b9cdda2332b326d2b6595577192099badff3c5b840f3
Nov 01 22:55:04.973226 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.973192    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:55:04.973331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.973266    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:55:04.973331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:04.973295    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:55:05.243562 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:05.243474    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" event={"ID":"8603b6a4-30a6-4b97-b2b9-2d0f80c74365","Type":"ContainerStarted","Data":"9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2"}
Nov 01 22:55:05.243562 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:05.243540    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" event={"ID":"8603b6a4-30a6-4b97-b2b9-2d0f80c74365","Type":"ContainerStarted","Data":"989091eb8d703b1a7e08b9cdda2332b326d2b6595577192099badff3c5b840f3"}
Nov 01 22:55:05.356870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:05.356807    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" podStartSLOduration=2.356789826 podStartE2EDuration="2.356789826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:05.353217196 +0000 UTC m=+1886.942416256" watchObservedRunningTime="2026-11-01 22:55:05.356789826 +0000 UTC m=+1886.945988891"
Nov 01 22:55:05.521115 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:05.515321    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:06.515269 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:06.515240    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:07.443064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.423866    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"]
Nov 01 22:55:07.477459 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.477428    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:07.477733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.477716    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:55:07.478228 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.478208    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:07.509394 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.508676    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"]
Nov 01 22:55:07.558373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.538747    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-584561-dockercfg-zm9vg\""
Nov 01 22:55:07.559807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.557722    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-584561\""
Nov 01 22:55:07.644080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.643395    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"]
Nov 01 22:55:07.726114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.723174    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8cc97-3f57-4e2f-95eb-148260e80ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft\" (UID: \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:07.726114 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.723327    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdxf\" (UniqueName: \"kubernetes.io/projected/73f8cc97-3f57-4e2f-95eb-148260e80ef9-kube-api-access-2pdxf\") pod \"payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft\" (UID: \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:07.808063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.802614    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"]
Nov 01 22:55:07.808063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.802741    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:07.904336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.890455    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2p4\" (UniqueName: \"kubernetes.io/projected/c493d14a-2273-48ad-9a2a-04e39b147078-kube-api-access-cj2p4\") pod \"payload-processing-e2e-api-b-584561-75964b6987-nm5tb\" (UID: \"c493d14a-2273-48ad-9a2a-04e39b147078\") " pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:07.904336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.902609    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8cc97-3f57-4e2f-95eb-148260e80ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft\" (UID: \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:07.904336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.902700    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493d14a-2273-48ad-9a2a-04e39b147078-plugins-config-volume\") pod \"payload-processing-e2e-api-b-584561-75964b6987-nm5tb\" (UID: \"c493d14a-2273-48ad-9a2a-04e39b147078\") " pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:07.904336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.902731    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdxf\" (UniqueName: \"kubernetes.io/projected/73f8cc97-3f57-4e2f-95eb-148260e80ef9-kube-api-access-2pdxf\") pod \"payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft\" (UID: \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:07.904336 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.904225    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8cc97-3f57-4e2f-95eb-148260e80ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft\" (UID: \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:07.957252 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:07.957213    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdxf\" (UniqueName: \"kubernetes.io/projected/73f8cc97-3f57-4e2f-95eb-148260e80ef9-kube-api-access-2pdxf\") pod \"payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft\" (UID: \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:08.003966 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.003897    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493d14a-2273-48ad-9a2a-04e39b147078-plugins-config-volume\") pod \"payload-processing-e2e-api-b-584561-75964b6987-nm5tb\" (UID: \"c493d14a-2273-48ad-9a2a-04e39b147078\") " pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:08.004134 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.004032    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2p4\" (UniqueName: \"kubernetes.io/projected/c493d14a-2273-48ad-9a2a-04e39b147078-kube-api-access-cj2p4\") pod \"payload-processing-e2e-api-b-584561-75964b6987-nm5tb\" (UID: \"c493d14a-2273-48ad-9a2a-04e39b147078\") " pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:08.004561 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.004543    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493d14a-2273-48ad-9a2a-04e39b147078-plugins-config-volume\") pod \"payload-processing-e2e-api-b-584561-75964b6987-nm5tb\" (UID: \"c493d14a-2273-48ad-9a2a-04e39b147078\") " pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:08.057294 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.057265    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2p4\" (UniqueName: \"kubernetes.io/projected/c493d14a-2273-48ad-9a2a-04e39b147078-kube-api-access-cj2p4\") pod \"payload-processing-e2e-api-b-584561-75964b6987-nm5tb\" (UID: \"c493d14a-2273-48ad-9a2a-04e39b147078\") " pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:08.206951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.206915    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:08.227486 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.227453    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:08.232601 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.232580    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 22:55:08.607570 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:55:08.607539    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f8cc97_3f57_4e2f_95eb_148260e80ef9.slice/crio-4f0e56c61f723eb7187a6d9ac5789dc67bf3acb2acf58e410fea8f3247175e76 WatchSource:0}: Error finding container 4f0e56c61f723eb7187a6d9ac5789dc67bf3acb2acf58e410fea8f3247175e76: Status 404 returned error can't find the container with id 4f0e56c61f723eb7187a6d9ac5789dc67bf3acb2acf58e410fea8f3247175e76
Nov 01 22:55:08.608829 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.608805    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"]
Nov 01 22:55:08.657946 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:08.657921    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"]
Nov 01 22:55:08.663075 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:55:08.663028    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc493d14a_2273_48ad_9a2a_04e39b147078.slice/crio-01cd1cdbb5e597b0920c9dee6fbe6c33c68af0034f11a6ee6066b6d9cd8853ff WatchSource:0}: Error finding container 01cd1cdbb5e597b0920c9dee6fbe6c33c68af0034f11a6ee6066b6d9cd8853ff: Status 404 returned error can't find the container with id 01cd1cdbb5e597b0920c9dee6fbe6c33c68af0034f11a6ee6066b6d9cd8853ff
Nov 01 22:55:09.505910 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:09.505874    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb" event={"ID":"c493d14a-2273-48ad-9a2a-04e39b147078","Type":"ContainerStarted","Data":"6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda"}
Nov 01 22:55:09.506247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:09.506221    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb" event={"ID":"c493d14a-2273-48ad-9a2a-04e39b147078","Type":"ContainerStarted","Data":"01cd1cdbb5e597b0920c9dee6fbe6c33c68af0034f11a6ee6066b6d9cd8853ff"}
Nov 01 22:55:09.506412 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:09.506381    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:09.508932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:09.508907    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft" event={"ID":"73f8cc97-3f57-4e2f-95eb-148260e80ef9","Type":"ContainerStarted","Data":"05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f"}
Nov 01 22:55:09.509056 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:09.508958    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft" event={"ID":"73f8cc97-3f57-4e2f-95eb-148260e80ef9","Type":"ContainerStarted","Data":"4f0e56c61f723eb7187a6d9ac5789dc67bf3acb2acf58e410fea8f3247175e76"}
Nov 01 22:55:09.509216 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:09.509200    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:09.613417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:09.613372    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb" podStartSLOduration=2.613361358 podStartE2EDuration="2.613361358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:09.61297141 +0000 UTC m=+1891.202170470" watchObservedRunningTime="2026-11-01 22:55:09.613361358 +0000 UTC m=+1891.202560419"
Nov 01 22:55:09.719553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:09.719500    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft" podStartSLOduration=2.7194875229999997 podStartE2EDuration="2.719487523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:09.71875976 +0000 UTC m=+1891.307958823" watchObservedRunningTime="2026-11-01 22:55:09.719487523 +0000 UTC m=+1891.308686575"
Nov 01 22:55:13.902229 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:13.902194    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:55:14.799449 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.799406    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"]
Nov 01 22:55:14.832382 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.832348    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:14.872138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.871031    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"]
Nov 01 22:55:14.900157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.898479    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f0206-dockercfg-hwt2j\""
Nov 01 22:55:14.900157 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.898544    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f0206\""
Nov 01 22:55:14.942117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.938349    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrj24\" (UniqueName: \"kubernetes.io/projected/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-kube-api-access-xrj24\") pod \"payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd\" (UID: \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:14.942117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.938494    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd\" (UID: \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:14.955026 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.954996    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"]
Nov 01 22:55:14.971752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:14.971728    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:15.300346 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.299234    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"]
Nov 01 22:55:15.373864 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.373818    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrj24\" (UniqueName: \"kubernetes.io/projected/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-kube-api-access-xrj24\") pod \"payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd\" (UID: \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:15.374065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.373990    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd\" (UID: \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:15.399012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.398977    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd\" (UID: \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:15.482066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.469864    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrj24\" (UniqueName: \"kubernetes.io/projected/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-kube-api-access-xrj24\") pod \"payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd\" (UID: \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:15.509185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.503111    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69561c0-80d8-4c4c-b924-b470b2c82a40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv\" (UID: \"d69561c0-80d8-4c4c-b924-b470b2c82a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:15.509185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.503204    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2cw\" (UniqueName: \"kubernetes.io/projected/d69561c0-80d8-4c4c-b924-b470b2c82a40-kube-api-access-bb2cw\") pod \"payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv\" (UID: \"d69561c0-80d8-4c4c-b924-b470b2c82a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:15.509185 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.503567    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:15.609316 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.609286    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69561c0-80d8-4c4c-b924-b470b2c82a40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv\" (UID: \"d69561c0-80d8-4c4c-b924-b470b2c82a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:15.610639 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.610618    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2cw\" (UniqueName: \"kubernetes.io/projected/d69561c0-80d8-4c4c-b924-b470b2c82a40-kube-api-access-bb2cw\") pod \"payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv\" (UID: \"d69561c0-80d8-4c4c-b924-b470b2c82a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:15.612429 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.610340    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69561c0-80d8-4c4c-b924-b470b2c82a40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv\" (UID: \"d69561c0-80d8-4c4c-b924-b470b2c82a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:15.727993 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.727888    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2cw\" (UniqueName: \"kubernetes.io/projected/d69561c0-80d8-4c4c-b924-b470b2c82a40-kube-api-access-bb2cw\") pod \"payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv\" (UID: \"d69561c0-80d8-4c4c-b924-b470b2c82a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:15.897390 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:15.897117    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:16.176109 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.176076    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"]
Nov 01 22:55:16.178193 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:55:16.178166    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3f58ca_e71f_4db3_bf28_c6fc5f5f3607.slice/crio-695b8c9c79b7eaaa66b65fba3af55c9e6c863d5bae096303783d0c3837a52a8e WatchSource:0}: Error finding container 695b8c9c79b7eaaa66b65fba3af55c9e6c863d5bae096303783d0c3837a52a8e: Status 404 returned error can't find the container with id 695b8c9c79b7eaaa66b65fba3af55c9e6c863d5bae096303783d0c3837a52a8e
Nov 01 22:55:16.372083 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.371526    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"]
Nov 01 22:55:16.372919 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:55:16.372895    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69561c0_80d8_4c4c_b924_b470b2c82a40.slice/crio-1cad1742e2ffafd3a27931fff8cf31c3e9c01d7bd4789c3d10a6aea71f192f1e WatchSource:0}: Error finding container 1cad1742e2ffafd3a27931fff8cf31c3e9c01d7bd4789c3d10a6aea71f192f1e: Status 404 returned error can't find the container with id 1cad1742e2ffafd3a27931fff8cf31c3e9c01d7bd4789c3d10a6aea71f192f1e
Nov 01 22:55:16.657391 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.657347    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv" event={"ID":"d69561c0-80d8-4c4c-b924-b470b2c82a40","Type":"ContainerStarted","Data":"e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43"}
Nov 01 22:55:16.657593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.657417    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv" event={"ID":"d69561c0-80d8-4c4c-b924-b470b2c82a40","Type":"ContainerStarted","Data":"1cad1742e2ffafd3a27931fff8cf31c3e9c01d7bd4789c3d10a6aea71f192f1e"}
Nov 01 22:55:16.657593 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.657466    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:16.659752 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.659724    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd" event={"ID":"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607","Type":"ContainerStarted","Data":"896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec"}
Nov 01 22:55:16.659877 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.659778    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd" event={"ID":"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607","Type":"ContainerStarted","Data":"695b8c9c79b7eaaa66b65fba3af55c9e6c863d5bae096303783d0c3837a52a8e"}
Nov 01 22:55:16.659877 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.659810    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:16.770174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.770035    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv" podStartSLOduration=2.770016884 podStartE2EDuration="2.770016884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:16.768303655 +0000 UTC m=+1898.357502716" watchObservedRunningTime="2026-11-01 22:55:16.770016884 +0000 UTC m=+1898.359215947"
Nov 01 22:55:16.923091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:16.918276    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd" podStartSLOduration=2.918257094 podStartE2EDuration="2.918257094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:16.881377663 +0000 UTC m=+1898.470576724" watchObservedRunningTime="2026-11-01 22:55:16.918257094 +0000 UTC m=+1898.507456156"
Nov 01 22:55:18.636064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:18.626403    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:55:18.714165 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:18.714119    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:55:18.765081 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:18.761745    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:55:18.839075 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:18.838145    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:55:18.993107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:18.988145    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:55:19.033264 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:55:19.033224    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:55:20.580855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:20.580814    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 22:55:20.582698 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:20.582674    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 22:55:22.768442 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.768400    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"]
Nov 01 22:55:22.788343 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.788315    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.842693 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.840857    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-103369-openshift-default-dockercfg-g987v\""
Nov 01 22:55:22.891539 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.891503    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"]
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907098    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-data\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907149    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-podinfo\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907271    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-envoy\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907340    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-certs\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907391    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istiod-ca-cert\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907511    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-token\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907540    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvhx\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-kube-api-access-dvvhx\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907634    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-credential-socket\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:22.913065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:22.907723    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-socket\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011252    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-data\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011305    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-podinfo\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011426    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-envoy\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011493    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-certs\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011546    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istiod-ca-cert\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011671    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-token\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011702    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvhx\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-kube-api-access-dvvhx\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011802    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-credential-socket\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.011895    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-socket\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.012485    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-socket\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.012746    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-data\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.014755    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istiod-ca-cert\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.015675 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.015403    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-certs\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.016500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.016188    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-credential-socket\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.055951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.055869    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-envoy\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.072289 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.072258    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvhx\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-kube-api-access-dvvhx\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.072748 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.072720    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-token\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.138861 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.138825    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-podinfo\") pod \"e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") " pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.139454 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.139428    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:23.634631 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.634436    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"]
Nov 01 22:55:23.672067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.654112    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:55:23.672515 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.672473    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:55:23.672765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.672740    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:55:23.890753 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:23.890723    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" event={"ID":"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad","Type":"ContainerStarted","Data":"5fc89396c4cdaaada90b6f6c03fe9b58fcd78e8b917dc6a85f212b7febefdec1"}
Nov 01 22:55:24.899569 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:24.899534    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" event={"ID":"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad","Type":"ContainerStarted","Data":"23bbea1a85e4fbb2e552da906c2e7d9dac078467d5ca1a75b8ee3cbf1ddb3825"}
Nov 01 22:55:25.060979 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:25.060929    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" podStartSLOduration=3.060915488 podStartE2EDuration="3.060915488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:25.058497809 +0000 UTC m=+1906.647696870" watchObservedRunningTime="2026-11-01 22:55:25.060915488 +0000 UTC m=+1906.650114548"
Nov 01 22:55:25.140280 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:25.140249    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:25.146483 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:25.146462    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:25.908462 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:25.908433    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:25.909528 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:25.909507    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:55:27.730063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:27.730019    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:55:27.736882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:27.736857    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:55:35.299067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.277295    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"]
Nov 01 22:55:35.451067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.433453    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"]
Nov 01 22:55:35.451067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.433586    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:35.515914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.515280    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbwb\" (UniqueName: \"kubernetes.io/projected/485db561-5b7b-4802-88c2-31ac2dce2e51-kube-api-access-dgbwb\") pod \"payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd\" (UID: \"485db561-5b7b-4802-88c2-31ac2dce2e51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:35.515914 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.515437    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485db561-5b7b-4802-88c2-31ac2dce2e51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd\" (UID: \"485db561-5b7b-4802-88c2-31ac2dce2e51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:35.554417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.554326    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-103369-dockercfg-srw7f\""
Nov 01 22:55:35.554695 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.554677    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-103369\""
Nov 01 22:55:35.616893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.616854    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbwb\" (UniqueName: \"kubernetes.io/projected/485db561-5b7b-4802-88c2-31ac2dce2e51-kube-api-access-dgbwb\") pod \"payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd\" (UID: \"485db561-5b7b-4802-88c2-31ac2dce2e51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:35.617117 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.616920    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485db561-5b7b-4802-88c2-31ac2dce2e51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd\" (UID: \"485db561-5b7b-4802-88c2-31ac2dce2e51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:35.617829 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.617804    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485db561-5b7b-4802-88c2-31ac2dce2e51-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd\" (UID: \"485db561-5b7b-4802-88c2-31ac2dce2e51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:35.670855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.670829    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbwb\" (UniqueName: \"kubernetes.io/projected/485db561-5b7b-4802-88c2-31ac2dce2e51-kube-api-access-dgbwb\") pod \"payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd\" (UID: \"485db561-5b7b-4802-88c2-31ac2dce2e51\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:35.794403 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:35.794371    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:36.167974 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:55:36.167945    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485db561_5b7b_4802_88c2_31ac2dce2e51.slice/crio-a31d6d8946717237788e01817dd0f1c1a591b2ba6d13500ffbd8df4fbce8d793 WatchSource:0}: Error finding container a31d6d8946717237788e01817dd0f1c1a591b2ba6d13500ffbd8df4fbce8d793: Status 404 returned error can't find the container with id a31d6d8946717237788e01817dd0f1c1a591b2ba6d13500ffbd8df4fbce8d793
Nov 01 22:55:36.168385 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:36.168365    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"]
Nov 01 22:55:37.071395 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:37.071358    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd" event={"ID":"485db561-5b7b-4802-88c2-31ac2dce2e51","Type":"ContainerStarted","Data":"c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7"}
Nov 01 22:55:37.071818 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:37.071458    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:37.071818 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:37.071479    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd" event={"ID":"485db561-5b7b-4802-88c2-31ac2dce2e51","Type":"ContainerStarted","Data":"a31d6d8946717237788e01817dd0f1c1a591b2ba6d13500ffbd8df4fbce8d793"}
Nov 01 22:55:37.181219 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:37.181163    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd" podStartSLOduration=2.181150081 podStartE2EDuration="2.181150081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:37.178527617 +0000 UTC m=+1918.767726678" watchObservedRunningTime="2026-11-01 22:55:37.181150081 +0000 UTC m=+1918.770349159"
Nov 01 22:55:39.499961 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:39.499927    4837 scope.go:122] "RemoveContainer" containerID="ea4429881ec83f97e274a9a8b739f8e6160e7c9c0a2e949c2c30d426cccb9bff"
Nov 01 22:55:46.089748 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.089713    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"]
Nov 01 22:55:46.090217 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.089942    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl" podUID="59837a37-e54d-4458-a7a3-15631f572036" containerName="maas-api" containerID="cri-o://126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746" gracePeriod=30
Nov 01 22:55:46.694112 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.691348    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:55:46.747963 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.747887    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls\") pod \"59837a37-e54d-4458-a7a3-15631f572036\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") "
Nov 01 22:55:46.748152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.748016    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59837a37-e54d-4458-a7a3-15631f572036-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59837a37-e54d-4458-a7a3-15631f572036-openshift-service-ca\") pod \"59837a37-e54d-4458-a7a3-15631f572036\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") "
Nov 01 22:55:46.748152 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.748074    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59837a37-e54d-4458-a7a3-15631f572036-kube-api-access-wsxw4\" (UniqueName: \"kubernetes.io/projected/59837a37-e54d-4458-a7a3-15631f572036-kube-api-access-wsxw4\") pod \"59837a37-e54d-4458-a7a3-15631f572036\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") "
Nov 01 22:55:46.748292 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.748251    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-metrics-certs\") pod \"59837a37-e54d-4458-a7a3-15631f572036\" (UID: \"59837a37-e54d-4458-a7a3-15631f572036\") "
Nov 01 22:55:46.748368 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.748339    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59837a37-e54d-4458-a7a3-15631f572036-openshift-service-ca" pod "59837a37-e54d-4458-a7a3-15631f572036" (UID: "59837a37-e54d-4458-a7a3-15631f572036"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:55:46.748967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.748941    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59837a37-e54d-4458-a7a3-15631f572036-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:55:46.750143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.750114    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls" pod "59837a37-e54d-4458-a7a3-15631f572036" (UID: "59837a37-e54d-4458-a7a3-15631f572036"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:55:46.750674 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.750607    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-metrics-certs" pod "59837a37-e54d-4458-a7a3-15631f572036" (UID: "59837a37-e54d-4458-a7a3-15631f572036"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:55:46.751059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.751023    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59837a37-e54d-4458-a7a3-15631f572036-kube-api-access-wsxw4" pod "59837a37-e54d-4458-a7a3-15631f572036" (UID: "59837a37-e54d-4458-a7a3-15631f572036"). InnerVolumeSpecName "kube-api-access-wsxw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:55:46.850958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.850926    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:55:46.850958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.850955    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59837a37-e54d-4458-a7a3-15631f572036-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:55:46.850958 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:46.850965    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsxw4\" (UniqueName: \"kubernetes.io/projected/59837a37-e54d-4458-a7a3-15631f572036-kube-api-access-wsxw4\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:55:47.212223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.209213    4837 generic.go:356] "Generic (PLEG): container finished" podID="59837a37-e54d-4458-a7a3-15631f572036" containerID="126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746" exitCode=0
Nov 01 22:55:47.212223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.209270    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl" event={"ID":"59837a37-e54d-4458-a7a3-15631f572036","Type":"ContainerDied","Data":"126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746"}
Nov 01 22:55:47.212223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.209324    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl" event={"ID":"59837a37-e54d-4458-a7a3-15631f572036","Type":"ContainerDied","Data":"647cbcc8be6abfb09f0a706abfbab87624ef28b2a9fca43ad05dbb844854694b"}
Nov 01 22:55:47.212223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.209352    4837 scope.go:122] "RemoveContainer" containerID="126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746"
Nov 01 22:55:47.212223 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.209537    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"
Nov 01 22:55:47.292330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.291416    4837 scope.go:122] "RemoveContainer" containerID="126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746"
Nov 01 22:55:47.292330 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:55:47.291873    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746\": container with ID starting with 126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746 not found: ID does not exist" containerID="126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746"
Nov 01 22:55:47.292330 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.291906    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746"} err="failed to get container status \"126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746\": rpc error: code = NotFound desc = could not find container \"126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746\": container with ID starting with 126fe6eb3e2eaeb465bef96faf4faf2c0cd28118dd48a84d671133c0e3e7a746 not found: ID does not exist"
Nov 01 22:55:47.404872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.404835    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"]
Nov 01 22:55:47.450901 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:47.450874    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-s4lcl"]
Nov 01 22:55:48.125621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:48.122688    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 22:55:48.526858 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:55:48.526827    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59837a37-e54d-4458-a7a3-15631f572036" path="/var/lib/kubelet/pods/59837a37-e54d-4458-a7a3-15631f572036/volumes"
Nov 01 22:55:48.815848 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:55:48.815761    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:56:01.218078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.206520    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"]
Nov 01 22:56:01.218078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.207600    4837 cpu_manager.go:414] "containerMap: removing container" podUID="59837a37-e54d-4458-a7a3-15631f572036" containerName="maas-api"
Nov 01 22:56:01.218078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.207620    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59837a37-e54d-4458-a7a3-15631f572036" containerName="maas-api"
Nov 01 22:56:01.218078 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.207816    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="59837a37-e54d-4458-a7a3-15631f572036" containerName="maas-api"
Nov 01 22:56:01.249502 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.246879    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"]
Nov 01 22:56:01.249502 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.246992    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:01.259443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.259405    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"]
Nov 01 22:56:01.266370 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.266345    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"]
Nov 01 22:56:01.266497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.266445    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:01.298813 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.298790    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8j5\" (UniqueName: \"kubernetes.io/projected/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-kube-api-access-2n8j5\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl\" (UID: \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:01.298972 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.298952    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrn4\" (UniqueName: \"kubernetes.io/projected/76032052-d23e-4753-9c1b-e3f59d504acf-kube-api-access-jmrn4\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9\" (UID: \"76032052-d23e-4753-9c1b-e3f59d504acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:01.299099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.299082    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl\" (UID: \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:01.299150 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.299117    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76032052-d23e-4753-9c1b-e3f59d504acf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9\" (UID: \"76032052-d23e-4753-9c1b-e3f59d504acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:01.300334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.300308    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-945efd02\""
Nov 01 22:56:01.346847 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.346823    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-945efd02-dockercfg-p2vtn\""
Nov 01 22:56:01.401037 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.401006    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrn4\" (UniqueName: \"kubernetes.io/projected/76032052-d23e-4753-9c1b-e3f59d504acf-kube-api-access-jmrn4\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9\" (UID: \"76032052-d23e-4753-9c1b-e3f59d504acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:01.401206 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.401184    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl\" (UID: \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:01.401281 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.401233    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76032052-d23e-4753-9c1b-e3f59d504acf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9\" (UID: \"76032052-d23e-4753-9c1b-e3f59d504acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:01.401405 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.401385    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8j5\" (UniqueName: \"kubernetes.io/projected/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-kube-api-access-2n8j5\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl\" (UID: \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:01.402057 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.402025    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76032052-d23e-4753-9c1b-e3f59d504acf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9\" (UID: \"76032052-d23e-4753-9c1b-e3f59d504acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:01.402163 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.402143    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl\" (UID: \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:01.455373 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.455351    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrn4\" (UniqueName: \"kubernetes.io/projected/76032052-d23e-4753-9c1b-e3f59d504acf-kube-api-access-jmrn4\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9\" (UID: \"76032052-d23e-4753-9c1b-e3f59d504acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:01.455481 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.455461    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8j5\" (UniqueName: \"kubernetes.io/projected/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-kube-api-access-2n8j5\") pod \"payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl\" (UID: \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\") " pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:01.568746 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.568668    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:01.584014 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.583988    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:01.957416 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:56:01.957390    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76032052_d23e_4753_9c1b_e3f59d504acf.slice/crio-48505e0574900db77b298d8c97e3ca552c96d780a63c547b6d1415bab7d65f86 WatchSource:0}: Error finding container 48505e0574900db77b298d8c97e3ca552c96d780a63c547b6d1415bab7d65f86: Status 404 returned error can't find the container with id 48505e0574900db77b298d8c97e3ca552c96d780a63c547b6d1415bab7d65f86
Nov 01 22:56:01.957602 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:01.957581    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"]
Nov 01 22:56:02.006140 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.006116    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"]
Nov 01 22:56:02.014810 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:56:02.014787    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1715f8e_51f0_4d86_80b3_dc6e43ea14dd.slice/crio-53afa1668b168f26635b7fb7d9827b93fb78b100a74b86644cd1e16441842dca WatchSource:0}: Error finding container 53afa1668b168f26635b7fb7d9827b93fb78b100a74b86644cd1e16441842dca: Status 404 returned error can't find the container with id 53afa1668b168f26635b7fb7d9827b93fb78b100a74b86644cd1e16441842dca
Nov 01 22:56:02.341747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.341652    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl" event={"ID":"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd","Type":"ContainerStarted","Data":"042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0"}
Nov 01 22:56:02.341747 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.341720    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl" event={"ID":"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd","Type":"ContainerStarted","Data":"53afa1668b168f26635b7fb7d9827b93fb78b100a74b86644cd1e16441842dca"}
Nov 01 22:56:02.342285 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.341754    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:02.343660 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.343632    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9" event={"ID":"76032052-d23e-4753-9c1b-e3f59d504acf","Type":"ContainerStarted","Data":"ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce"}
Nov 01 22:56:02.343801 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.343680    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9" event={"ID":"76032052-d23e-4753-9c1b-e3f59d504acf","Type":"ContainerStarted","Data":"48505e0574900db77b298d8c97e3ca552c96d780a63c547b6d1415bab7d65f86"}
Nov 01 22:56:02.343801 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.343722    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:02.452358 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.452261    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl" podStartSLOduration=2.452242931 podStartE2EDuration="2.452242931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:02.448595188 +0000 UTC m=+1944.037794249" watchObservedRunningTime="2026-11-01 22:56:02.452242931 +0000 UTC m=+1944.041441993"
Nov 01 22:56:02.557237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:02.557162    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9" podStartSLOduration=2.5571444960000003 podStartE2EDuration="2.557144496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:02.556879253 +0000 UTC m=+1944.146078315" watchObservedRunningTime="2026-11-01 22:56:02.557144496 +0000 UTC m=+1944.146343558"
Nov 01 22:56:05.372980 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:05.372911    4837 generic.go:356] "Generic (PLEG): container finished" podID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerID="31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd" exitCode=0
Nov 01 22:56:05.373375 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:05.372980    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" event={"ID":"c510ab0f-4027-4c03-aead-96adc4e1cb3a","Type":"ContainerDied","Data":"31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd"}
Nov 01 22:56:05.373417 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:05.373392    4837 scope.go:122] "RemoveContainer" containerID="31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd"
Nov 01 22:56:05.375190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:05.375157    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:56:05.375301 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:05.375242    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:56:05.375301 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:05.375290    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:56:06.382800 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:06.382766    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" event={"ID":"c510ab0f-4027-4c03-aead-96adc4e1cb3a","Type":"ContainerStarted","Data":"361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b"}
Nov 01 22:56:07.204014 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:07.203979    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:56:07.204931 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:07.204909    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:56:07.205079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:07.204955    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:56:08.204236 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:08.204201    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:56:08.204733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:08.204275    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:56:09.204520 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:09.204486    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:56:09.204927 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:09.204551    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:56:10.204153 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:10.204113    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:56:10.204356 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:10.204183    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:56:11.205125 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:11.205090    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:56:11.205617 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:11.205160    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:56:12.210487 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:12.210449    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:56:12.210981 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:12.210966    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:56:12.211635 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:12.211621    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 22:56:13.353764 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:13.353736    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:56:13.354171 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:13.353806    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:56:17.678856 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:17.678824    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:56:18.818169 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:56:18.818134    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:56:29.015249 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.015217    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"]
Nov 01 22:56:29.021445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.021425    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.108184 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.108152    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dww4r\" (UniqueName: \"kubernetes.io/projected/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-kube-api-access-dww4r\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.108347 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.108314    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.108469 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.108445    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-openshift-service-ca\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.108520 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.108509    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-metrics-certs\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.118133 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.118110    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-584561-serving-cert\""
Nov 01 22:56:29.124825 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.124799    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"]
Nov 01 22:56:29.209854 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.209822    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-openshift-service-ca\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.210059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.209879    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-metrics-certs\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.210059 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.209987    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dww4r\" (UniqueName: \"kubernetes.io/projected/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-kube-api-access-dww4r\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.210205 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.210099    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.210292 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:56:29.210234    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-584561-serving-cert: secret "maas-api-e2e-api-b-584561-serving-cert" not found
Nov 01 22:56:29.210353 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:56:29.210303    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls podName:d4f17758-fa93-4a5b-b5a7-ff38029a3ab2 nodeName:}" failed. No retries permitted until 2026-11-01 22:56:29.710283019 +0000 UTC m=+1971.299482072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls") pod "maas-api-e2e-api-b-584561-f985548c7-gmd5c" (UID: "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2") : secret "maas-api-e2e-api-b-584561-serving-cert" not found
Nov 01 22:56:29.210600 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.210579    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-openshift-service-ca\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.212263 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.212246    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-metrics-certs\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.266156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.266017    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dww4r\" (UniqueName: \"kubernetes.io/projected/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-kube-api-access-dww4r\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.718244 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.718209    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.720697 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.720673    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls\") pod \"maas-api-e2e-api-b-584561-f985548c7-gmd5c\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:29.957437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:29.957399    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:30.357495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:30.357317    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"]
Nov 01 22:56:30.699456 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:30.699419    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c" event={"ID":"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2","Type":"ContainerStarted","Data":"a8f19fc1b133f3d9f31e28b8c2d0e367bb629f90930825c42001dbbc8f214a29"}
Nov 01 22:56:31.710116 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:31.710036    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c" event={"ID":"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2","Type":"ContainerStarted","Data":"57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17"}
Nov 01 22:56:31.710497 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:31.710265    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:31.818534 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:31.818474    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c" podStartSLOduration=2.8415729069999998 podStartE2EDuration="3.818456179s" totalImagesPullingTime="976.883272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:31.817626378 +0000 UTC m=+1973.406825438" watchObservedRunningTime="2026-11-01 22:56:31.818456179 +0000 UTC m=+1973.407655244"
Nov 01 22:56:37.722586 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:37.722550    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 22:56:38.770351 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:38.770317    4837 generic.go:356] "Generic (PLEG): container finished" podID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerID="6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954" exitCode=0
Nov 01 22:56:38.770778 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:38.770386    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" event={"ID":"cf0049fa-0562-4103-8df1-ba15d48f58ba","Type":"ContainerDied","Data":"6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954"}
Nov 01 22:56:38.770872 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:38.770797    4837 scope.go:122] "RemoveContainer" containerID="6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954"
Nov 01 22:56:38.772565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:38.772536    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:56:38.772638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:38.772618    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:56:38.772687 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:38.772650    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:56:39.780851 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:39.780811    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" event={"ID":"cf0049fa-0562-4103-8df1-ba15d48f58ba","Type":"ContainerStarted","Data":"69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a"}
Nov 01 22:56:40.669945 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:40.669915    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:56:40.671080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:40.671033    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:56:40.671234 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:40.671106    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:56:41.676066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:41.675477    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:56:41.676066 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:41.675558    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:56:42.683067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:42.682441    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:56:42.683067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:42.682517    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:56:42.736070 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:42.735367    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 22:56:42.825174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:42.825123    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:42.826258 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:42.826228    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:42.976120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:42.953685    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:42.976120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:42.954922    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:42.999100 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:42.988492    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:43.211475 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.211425    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:43.274237 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.274126    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:43.301356 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.301309    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:43.340174 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.340138    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:56:43.471139 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.470177    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 22:56:43.520728 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.520691    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:43.726063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.725941    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:56:43.726063 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.726018    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:56:43.776068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.757255    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:56:43.776068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.763063    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:43.776068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.764232    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:56:43.802733 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.802167    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:43.840067 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.839035    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:43.840267 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.840252    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:56:43.978312 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.968836    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:56:43.978850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:43.978814    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 01 22:56:44.068095 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.044419    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:56:44.068095 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.044498    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:56:44.095065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.089181    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:44.357771 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.330100    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:44.358611 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.358068    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:56:44.526077 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.525150    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:44.737017 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.736985    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:56:44.738749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.738717    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:44.774887 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.774847    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:56:44.784451 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.784405    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:44.784823 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.784725    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:56:44.830090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.818119    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.132:8000/ready\": context deadline exceeded"
Nov 01 22:56:44.903478 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.903426    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 01 22:56:44.904423 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.904396    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:56:44.904608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.904589    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:56:44.994001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.974274    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:56:44.994001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.975406    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 22:56:44.994001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.989161    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 22:56:44.994001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:44.989791    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 22:56:45.001960 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:45.001940    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 22:56:45.003465 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:45.003445    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 22:56:45.018860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:45.017456    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 22:56:45.018860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:45.018153    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 22:56:45.664643 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:45.664607    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:56:45.664860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:45.664659    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:56:46.672156 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:46.672111    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:56:46.674247 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:46.674223    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 22:56:46.999023 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:46.998946    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"]
Nov 01 22:56:46.999360 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:46.999328    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy" containerID="cri-o://5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72" gracePeriod=30
Nov 01 22:56:47.139296 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:47.139260    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"]
Nov 01 22:56:47.139503 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:47.139452    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy" containerID="cri-o://f59eeb55b1c0ecd90a6c9648a4c07a6303bc12a8e3eca3d83b64cf2448d4dd35" gracePeriod=30
Nov 01 22:56:48.815187 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:56:48.814679    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:56:52.300365 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.300334    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.339397    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-istio-token\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-istio-token\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.339840    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-data\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.339949    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-certs\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.340000    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-socket\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.340987    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-data" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.341109    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-socket" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.341141    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a66841d-00fa-47c5-a555-573984719a34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a66841d-00fa-47c5-a555-573984719a34-istio-podinfo\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.341229    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-kube-api-access-w8k6v\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-kube-api-access-w8k6v\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.341328    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-envoy\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.341358    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a66841d-00fa-47c5-a555-573984719a34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a66841d-00fa-47c5-a555-573984719a34-istiod-ca-cert\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.341433    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-credential-socket\") pod \"0a66841d-00fa-47c5-a555-573984719a34\" (UID: \"0a66841d-00fa-47c5-a555-573984719a34\") "
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.342565    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a66841d-00fa-47c5-a555-573984719a34-istiod-ca-cert" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.343512    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-credential-socket" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.344094    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a66841d-00fa-47c5-a555-573984719a34-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.344117    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.344132    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.354892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.344145    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.357594 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.357548    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-certs" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.371291 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.371253    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a66841d-00fa-47c5-a555-573984719a34-istio-podinfo" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:52.376511 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.374637    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-kube-api-access-w8k6v" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "kube-api-access-w8k6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.376511 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.375814    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-istio-token" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.380654 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.380156    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-envoy" pod "0a66841d-00fa-47c5-a555-573984719a34" (UID: "0a66841d-00fa-47c5-a555-573984719a34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.381153 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.381090    4837 generic.go:356] "Generic (PLEG): container finished" podID="0a66841d-00fa-47c5-a555-573984719a34" containerID="5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72" exitCode=0
Nov 01 22:56:52.381302 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.381248    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" event={"ID":"0a66841d-00fa-47c5-a555-573984719a34","Type":"ContainerDied","Data":"5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72"}
Nov 01 22:56:52.381380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.381350    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92" event={"ID":"0a66841d-00fa-47c5-a555-573984719a34","Type":"ContainerDied","Data":"6104327f17df23de1bb8c03afb1b5862853e7165a6f51077432a4d25dc476996"}
Nov 01 22:56:52.381445 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.381397    4837 scope.go:122] "RemoveContainer" containerID="5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72"
Nov 01 22:56:52.382090 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.381587    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"
Nov 01 22:56:52.394332 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.394175    4837 generic.go:356] "Generic (PLEG): container finished" podID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerID="f59eeb55b1c0ecd90a6c9648a4c07a6303bc12a8e3eca3d83b64cf2448d4dd35" exitCode=0
Nov 01 22:56:52.394332 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.394239    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" event={"ID":"2d92b6c6-a95e-4486-ab06-067af8b1d065","Type":"ContainerDied","Data":"f59eeb55b1c0ecd90a6c9648a4c07a6303bc12a8e3eca3d83b64cf2448d4dd35"}
Nov 01 22:56:52.405951 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.405924    4837 scope.go:122] "RemoveContainer" containerID="5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72"
Nov 01 22:56:52.406333 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:56:52.406309    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72\": container with ID starting with 5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72 not found: ID does not exist" containerID="5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72"
Nov 01 22:56:52.406472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.406344    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72"} err="failed to get container status \"5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72\": rpc error: code = NotFound desc = could not find container \"5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72\": container with ID starting with 5e773ce06c909dc20930eb2b5a633d747264be36c1cbc835bbc3353f920acf72 not found: ID does not exist"
Nov 01 22:56:52.443513 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.443490    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:56:52.455242 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.455220    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.455334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.455243    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a66841d-00fa-47c5-a555-573984719a34-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.455334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.455254    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8k6v\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-kube-api-access-w8k6v\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.455334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.455264    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a66841d-00fa-47c5-a555-573984719a34-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.455334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.455274    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a66841d-00fa-47c5-a555-573984719a34-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.556885 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.556800    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-certs\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.556885 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.556835    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-data\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.556885 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.556879    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-kube-api-access-xv67p\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-kube-api-access-xv67p\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.557197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.556927    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-token\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-token\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.557197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.556978    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d92b6c6-a95e-4486-ab06-067af8b1d065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d92b6c6-a95e-4486-ab06-067af8b1d065-istiod-ca-cert\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.557197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557010    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-socket\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.557197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557107    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-podinfo\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.557197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557185    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-credential-socket\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.557197 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557163    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-certs" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.557503 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557227    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-data" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.557503 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557370    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-envoy\") pod \"2d92b6c6-a95e-4486-ab06-067af8b1d065\" (UID: \"2d92b6c6-a95e-4486-ab06-067af8b1d065\") "
Nov 01 22:56:52.557503 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557407    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-socket" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.557503 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557465    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-credential-socket" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.557709 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.557525    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d92b6c6-a95e-4486-ab06-067af8b1d065-istiod-ca-cert" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.558662    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.558691    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.558705    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d92b6c6-a95e-4486-ab06-067af8b1d065-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.558718    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.558733    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.559800    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-podinfo" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.559930    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-kube-api-access-xv67p" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "kube-api-access-xv67p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.560597    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-token" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.560926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.560673    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-envoy" pod "2d92b6c6-a95e-4486-ab06-067af8b1d065" (UID: "2d92b6c6-a95e-4486-ab06-067af8b1d065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.623578 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.623550    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"]
Nov 01 22:56:52.628334 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.628313    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-zsg92"]
Nov 01 22:56:52.660769 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.660739    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.660769 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.660767    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.660769 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.660776    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv67p\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-kube-api-access-xv67p\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.661106 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:52.660786    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d92b6c6-a95e-4486-ab06-067af8b1d065-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.404437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:53.404355    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c" event={"ID":"2d92b6c6-a95e-4486-ab06-067af8b1d065","Type":"ContainerDied","Data":"4ee2d54596090c35dc9524e8145e2d3c3749094c8bd249fd9cfb7e6bd8eba710"}
Nov 01 22:56:53.404437 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:53.404409    4837 scope.go:122] "RemoveContainer" containerID="f59eeb55b1c0ecd90a6c9648a4c07a6303bc12a8e3eca3d83b64cf2448d4dd35"
Nov 01 22:56:53.404967 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:53.404368    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"
Nov 01 22:56:53.565664 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:53.565629    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"]
Nov 01 22:56:53.621119 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:53.618519    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p298c"]
Nov 01 22:56:54.640064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:54.633221    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a66841d-00fa-47c5-a555-573984719a34" path="/var/lib/kubelet/pods/0a66841d-00fa-47c5-a555-573984719a34/volumes"
Nov 01 22:56:54.640064 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:56:54.635775    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" path="/var/lib/kubelet/pods/2d92b6c6-a95e-4486-ab06-067af8b1d065/volumes"
Nov 01 22:57:10.764151 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:10.763857    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:57:18.825808 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:57:18.825768    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:57:22.258552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.258518    4837 generic.go:356] "Generic (PLEG): container finished" podID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerID="9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2" exitCode=0
Nov 01 22:57:22.259132 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.258595    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" event={"ID":"8603b6a4-30a6-4b97-b2b9-2d0f80c74365","Type":"ContainerDied","Data":"9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2"}
Nov 01 22:57:22.259207 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.259193    4837 scope.go:122] "RemoveContainer" containerID="9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2"
Nov 01 22:57:22.260395 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.260376    4837 generic.go:356] "Generic (PLEG): container finished" podID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerID="3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846" exitCode=0
Nov 01 22:57:22.260477 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.260467    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" event={"ID":"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4","Type":"ContainerDied","Data":"3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846"}
Nov 01 22:57:22.260844 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.260823    4837 scope.go:122] "RemoveContainer" containerID="3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846"
Nov 01 22:57:22.261222 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.261192    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:57:22.261303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.261274    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:57:22.261395 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.261341    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:57:22.262602 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.262572    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:57:22.262671 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.262656    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:57:22.262712 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.262703    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:57:22.468926 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:22.468899    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:57:23.270440 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:23.270402    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" event={"ID":"8603b6a4-30a6-4b97-b2b9-2d0f80c74365","Type":"ContainerStarted","Data":"0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2"}
Nov 01 22:57:23.272532 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:23.272506    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" event={"ID":"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4","Type":"ContainerStarted","Data":"05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310"}
Nov 01 22:57:23.510685 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:23.510648    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:57:23.512187 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:23.512155    4837 patch_prober.go:29] interesting pod/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:57:23.512344 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:23.512224    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 01 22:57:23.770217 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:23.770111    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:57:23.770435 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:23.770297    4837 patch_prober.go:29] interesting pod/e2e-api-b-584561-openshift-default-95c6887c-vdglb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:57:23.770435 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:23.770349    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:57:24.509754 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:24.509715    4837 patch_prober.go:29] interesting pod/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:57:24.510241 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:24.509763    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 01 22:57:24.769759 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:24.769670    4837 patch_prober.go:29] interesting pod/e2e-api-b-584561-openshift-default-95c6887c-vdglb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:57:24.770030 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:24.769998    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:57:25.509608 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:25.509560    4837 patch_prober.go:29] interesting pod/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:57:25.509799 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:25.509624    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 01 22:57:25.774126 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:25.770846    4837 patch_prober.go:29] interesting pod/e2e-api-b-584561-openshift-default-95c6887c-vdglb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:57:25.774126 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:25.770904    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:57:26.509667 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:26.509634    4837 patch_prober.go:29] interesting pod/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:57:26.509850 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:26.509690    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 01 22:57:26.786120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:26.776666    4837 patch_prober.go:29] interesting pod/e2e-api-b-584561-openshift-default-95c6887c-vdglb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:57:26.786120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:26.776727    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:57:27.509902 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:27.509865    4837 patch_prober.go:29] interesting pod/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:57:27.510322 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:27.509917    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 01 22:57:27.786400 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:27.786216    4837 patch_prober.go:29] interesting pod/e2e-api-b-584561-openshift-default-95c6887c-vdglb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:57:27.786400 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:27.786279    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:57:28.509539 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:28.509502    4837 patch_prober.go:29] interesting pod/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:57:28.509721 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:28.509565    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 01 22:57:28.769452 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:28.769350    4837 patch_prober.go:29] interesting pod/e2e-api-b-584561-openshift-default-95c6887c-vdglb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:57:28.769452 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:28.769417    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:57:29.515637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:29.515602    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:57:29.516054 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:29.516017    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:57:29.516744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:29.516725    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 22:57:29.775020 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:29.774944    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:57:29.775451 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:29.775249    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:57:29.775973 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:57:29.775957    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 22:57:48.813703 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:57:48.813668    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:58:02.772190 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:02.772154    4837 generic.go:356] "Generic (PLEG): container finished" podID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerID="23bbea1a85e4fbb2e552da906c2e7d9dac078467d5ca1a75b8ee3cbf1ddb3825" exitCode=0
Nov 01 22:58:02.772607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:02.772226    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" event={"ID":"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad","Type":"ContainerDied","Data":"23bbea1a85e4fbb2e552da906c2e7d9dac078467d5ca1a75b8ee3cbf1ddb3825"}
Nov 01 22:58:02.772649 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:02.772623    4837 scope.go:122] "RemoveContainer" containerID="23bbea1a85e4fbb2e552da906c2e7d9dac078467d5ca1a75b8ee3cbf1ddb3825"
Nov 01 22:58:02.774481 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:02.774457    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:58:02.774552 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:02.774526    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:58:02.774590 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:02.774556    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:58:03.781681 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:03.781646    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" event={"ID":"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad","Type":"ContainerStarted","Data":"3ed33fbbe2633d7da985d8f65f031b6fa0968c8e6963e57cd399d6ad3dd4a37f"}
Nov 01 22:58:04.142533 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:04.142448    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:58:04.142882 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:04.142795    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 22:58:04.143086 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:04.143061    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 22:58:05.153068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:05.150400    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 22:58:05.153068 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:05.150471    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 22:58:06.155431 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:06.155392    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 22:58:06.155849 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:06.155460    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 22:58:07.153065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:07.152443    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 22:58:07.153065 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:07.152519    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 22:58:08.146424 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.146390    4837 patch_prober.go:29] interesting pod/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 22:58:08.147018 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.146986    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 22:58:08.159809 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.159780    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"]
Nov 01 22:58:08.160516 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.160501    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy"
Nov 01 22:58:08.160575 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.160517    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy"
Nov 01 22:58:08.160612 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.160597    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy"
Nov 01 22:58:08.160612 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.160603    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy"
Nov 01 22:58:08.160715 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.160706    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d92b6c6-a95e-4486-ab06-067af8b1d065" containerName="istio-proxy"
Nov 01 22:58:08.160751 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.160718    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a66841d-00fa-47c5-a555-573984719a34" containerName="istio-proxy"
Nov 01 22:58:08.163894 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.163876    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.222731 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.222696    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.222860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.222737    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.222860 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.222840    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.223120 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.223101    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.223209 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.223138    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qf4t\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-kube-api-access-8qf4t\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.223209 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.223165    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.223324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.223223    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.223383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.223343    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.223383 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.223376    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324350 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324317    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324517 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324485    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324517 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324507    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qf4t\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-kube-api-access-8qf4t\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324606 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324524    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324606 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324541    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324606 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324590    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324607    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324649    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324670    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.324765 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324729    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.325001 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.324880    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.325124 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.325096    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.325260 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.325165    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.325607 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.325589    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.326942 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.326920    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.327148 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.327131    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.367954 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.367920    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"]
Nov 01 22:58:08.381303 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.381277    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.383054 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.382989    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qf4t\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-kube-api-access-8qf4t\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.479335 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.479304    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:08.924079 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:58:08.921906    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979e8cc3_6842_4ea6_98cd_4c6c4ac52495.slice/crio-8ab5cfee830f60731e0078ff60c4853da8f036dce3c96470801869693fa40278 WatchSource:0}: Error finding container 8ab5cfee830f60731e0078ff60c4853da8f036dce3c96470801869693fa40278: Status 404 returned error can't find the container with id 8ab5cfee830f60731e0078ff60c4853da8f036dce3c96470801869693fa40278
Nov 01 22:58:08.924079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.922518    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"]
Nov 01 22:58:08.925807 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.925774    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:58:08.925892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.925840    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:58:08.925892 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:08.925867    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 22:58:09.145973 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:09.145943    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:58:09.146481 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:09.146458    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:58:09.147351 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:09.147334    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 22:58:09.859255 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:09.859200    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" event={"ID":"979e8cc3-6842-4ea6-98cd-4c6c4ac52495","Type":"ContainerStarted","Data":"fd9766dddd371d31dd4fe13be28a5cced98c90874be3cc439f86eab3cbb0870f"}
Nov 01 22:58:09.859468 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:09.859286    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" event={"ID":"979e8cc3-6842-4ea6-98cd-4c6c4ac52495","Type":"ContainerStarted","Data":"8ab5cfee830f60731e0078ff60c4853da8f036dce3c96470801869693fa40278"}
Nov 01 22:58:09.973666 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:09.973593    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" podStartSLOduration=1.97357416 podStartE2EDuration="1.97357416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:09.969595971 +0000 UTC m=+2071.558795032" watchObservedRunningTime="2026-11-01 22:58:09.97357416 +0000 UTC m=+2071.562773221"
Nov 01 22:58:10.480158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:10.480124    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:10.482138 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:10.482113    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 01 22:58:10.482287 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:10.482166    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 01 22:58:11.480642 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:11.480610    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 01 22:58:11.481101 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:11.480661    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 01 22:58:12.479803 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:12.479766    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 01 22:58:12.480013 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:12.479896    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 01 22:58:13.480331 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:13.480294    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 01 22:58:13.480749 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:13.480357    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 01 22:58:14.480091 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:14.480024    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 01 22:58:14.480302 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:14.480120    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 01 22:58:15.480498 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:15.480466    4837 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 01 22:58:15.480885 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:15.480532    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 01 22:58:15.875198 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:15.875116    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:58:16.485173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:16.485142    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:16.485650 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:16.485587    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:16.486279 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:16.486257    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 22:58:18.815824 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:58:18.815786    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:58:23.961233 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:23.961196    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"]
Nov 01 22:58:24.068354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:24.068323    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"]
Nov 01 22:58:28.961661 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:28.961626    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv" podUID="d69561c0-80d8-4c4c-b924-b470b2c82a40" containerName="payload-processing" containerID="cri-o://e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43" gracePeriod=25
Nov 01 22:58:29.068797 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.068764    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd" podUID="db3f58ca-e71f-4db3-bf28-c6fc5f5f3607" containerName="payload-pre-processing" containerID="cri-o://896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec" gracePeriod=25
Nov 01 22:58:29.261455 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.261434    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:58:29.369744 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.369715    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:58:29.386978 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.386950    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69561c0-80d8-4c4c-b924-b470b2c82a40-kube-api-access-bb2cw\" (UniqueName: \"kubernetes.io/projected/d69561c0-80d8-4c4c-b924-b470b2c82a40-kube-api-access-bb2cw\") pod \"d69561c0-80d8-4c4c-b924-b470b2c82a40\" (UID: \"d69561c0-80d8-4c4c-b924-b470b2c82a40\") "
Nov 01 22:58:29.387324 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.387290    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69561c0-80d8-4c4c-b924-b470b2c82a40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69561c0-80d8-4c4c-b924-b470b2c82a40-plugins-config-volume\") pod \"d69561c0-80d8-4c4c-b924-b470b2c82a40\" (UID: \"d69561c0-80d8-4c4c-b924-b470b2c82a40\") "
Nov 01 22:58:29.387735 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.387712    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69561c0-80d8-4c4c-b924-b470b2c82a40-plugins-config-volume" pod "d69561c0-80d8-4c4c-b924-b470b2c82a40" (UID: "d69561c0-80d8-4c4c-b924-b470b2c82a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:29.388122 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.388100    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69561c0-80d8-4c4c-b924-b470b2c82a40-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:29.389644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.389616    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69561c0-80d8-4c4c-b924-b470b2c82a40-kube-api-access-bb2cw" pod "d69561c0-80d8-4c4c-b924-b470b2c82a40" (UID: "d69561c0-80d8-4c4c-b924-b470b2c82a40"). InnerVolumeSpecName "kube-api-access-bb2cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:29.489615 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.489543    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-plugins-config-volume\") pod \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\" (UID: \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\") "
Nov 01 22:58:29.489767 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.489752    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-kube-api-access-xrj24\" (UniqueName: \"kubernetes.io/projected/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-kube-api-access-xrj24\") pod \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\" (UID: \"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607\") "
Nov 01 22:58:29.489838 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.489808    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-plugins-config-volume" pod "db3f58ca-e71f-4db3-bf28-c6fc5f5f3607" (UID: "db3f58ca-e71f-4db3-bf28-c6fc5f5f3607"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:29.490277 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.490258    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:29.490277 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.490277    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb2cw\" (UniqueName: \"kubernetes.io/projected/d69561c0-80d8-4c4c-b924-b470b2c82a40-kube-api-access-bb2cw\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:29.491870 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.491840    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-kube-api-access-xrj24" pod "db3f58ca-e71f-4db3-bf28-c6fc5f5f3607" (UID: "db3f58ca-e71f-4db3-bf28-c6fc5f5f3607"). InnerVolumeSpecName "kube-api-access-xrj24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:29.591934 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:29.591897    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrj24\" (UniqueName: \"kubernetes.io/projected/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607-kube-api-access-xrj24\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:30.085815 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.085780    4837 generic.go:356] "Generic (PLEG): container finished" podID="db3f58ca-e71f-4db3-bf28-c6fc5f5f3607" containerID="896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec" exitCode=0
Nov 01 22:58:30.086359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.085857    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"
Nov 01 22:58:30.086359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.085866    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd" event={"ID":"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607","Type":"ContainerDied","Data":"896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec"}
Nov 01 22:58:30.086359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.085948    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd" event={"ID":"db3f58ca-e71f-4db3-bf28-c6fc5f5f3607","Type":"ContainerDied","Data":"695b8c9c79b7eaaa66b65fba3af55c9e6c863d5bae096303783d0c3837a52a8e"}
Nov 01 22:58:30.086359 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.085979    4837 scope.go:122] "RemoveContainer" containerID="896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec"
Nov 01 22:58:30.087436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.087341    4837 generic.go:356] "Generic (PLEG): container finished" podID="d69561c0-80d8-4c4c-b924-b470b2c82a40" containerID="e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43" exitCode=0
Nov 01 22:58:30.087436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.087403    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv" event={"ID":"d69561c0-80d8-4c4c-b924-b470b2c82a40","Type":"ContainerDied","Data":"e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43"}
Nov 01 22:58:30.087436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.087420    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"
Nov 01 22:58:30.087436 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.087434    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv" event={"ID":"d69561c0-80d8-4c4c-b924-b470b2c82a40","Type":"ContainerDied","Data":"1cad1742e2ffafd3a27931fff8cf31c3e9c01d7bd4789c3d10a6aea71f192f1e"}
Nov 01 22:58:30.099703 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.099679    4837 scope.go:122] "RemoveContainer" containerID="896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec"
Nov 01 22:58:30.100002 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:58:30.099981    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec\": container with ID starting with 896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec not found: ID does not exist" containerID="896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec"
Nov 01 22:58:30.100102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.100010    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec"} err="failed to get container status \"896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec\": rpc error: code = NotFound desc = could not find container \"896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec\": container with ID starting with 896ecbe35c8c4ffab8a0867b0566296b831b4e5baee405fe1f93ee2f61b9e8ec not found: ID does not exist"
Nov 01 22:58:30.100102 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.100026    4837 scope.go:122] "RemoveContainer" containerID="e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43"
Nov 01 22:58:30.109682 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.109665    4837 scope.go:122] "RemoveContainer" containerID="e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43"
Nov 01 22:58:30.109964 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:58:30.109943    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43\": container with ID starting with e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43 not found: ID does not exist" containerID="e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43"
Nov 01 22:58:30.110079 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.109968    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43"} err="failed to get container status \"e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43\": rpc error: code = NotFound desc = could not find container \"e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43\": container with ID starting with e57117311f89854be89522a7a081722004723adce1398d2b6da1715f91d1ec43 not found: ID does not exist"
Nov 01 22:58:30.241309 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.241227    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"]
Nov 01 22:58:30.287180 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.287150    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f0206-9c9668575-tddvv"]
Nov 01 22:58:30.388688 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.388657    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"]
Nov 01 22:58:30.392508 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.392486    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f0206-5c858b7647-ps2qd"]
Nov 01 22:58:30.528676 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.528605    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69561c0-80d8-4c4c-b924-b470b2c82a40" path="/var/lib/kubelet/pods/d69561c0-80d8-4c4c-b924-b470b2c82a40/volumes"
Nov 01 22:58:30.529088 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:30.529074    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3f58ca-e71f-4db3-bf28-c6fc5f5f3607" path="/var/lib/kubelet/pods/db3f58ca-e71f-4db3-bf28-c6fc5f5f3607/volumes"
Nov 01 22:58:35.337361 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.337326    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"]
Nov 01 22:58:35.338143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.338114    4837 cpu_manager.go:414] "containerMap: removing container" podUID="db3f58ca-e71f-4db3-bf28-c6fc5f5f3607" containerName="payload-pre-processing"
Nov 01 22:58:35.338143 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.338130    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3f58ca-e71f-4db3-bf28-c6fc5f5f3607" containerName="payload-pre-processing"
Nov 01 22:58:35.338313 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.338215    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d69561c0-80d8-4c4c-b924-b470b2c82a40" containerName="payload-processing"
Nov 01 22:58:35.338313 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.338221    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69561c0-80d8-4c4c-b924-b470b2c82a40" containerName="payload-processing"
Nov 01 22:58:35.338418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.338335    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69561c0-80d8-4c4c-b924-b470b2c82a40" containerName="payload-processing"
Nov 01 22:58:35.338418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.338345    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3f58ca-e71f-4db3-bf28-c6fc5f5f3607" containerName="payload-pre-processing"
Nov 01 22:58:35.343595 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.343577    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.392556 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.392531    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d68bab-serving-cert\""
Nov 01 22:58:35.445099 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.445068    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"]
Nov 01 22:58:35.473286 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.473254    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx997\" (UniqueName: \"kubernetes.io/projected/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-kube-api-access-mx997\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.473472 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.473436    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-metrics-certs\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.473606 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.473580    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-maas-api-tls\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.473904 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.473877    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.575493 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.575465    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.575658 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.575536    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx997\" (UniqueName: \"kubernetes.io/projected/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-kube-api-access-mx997\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.575702 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.575678    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-metrics-certs\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.575757 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.575742    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-maas-api-tls\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.576255 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.576237    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.578104 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.578082    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-metrics-certs\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.578282 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.578261    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-maas-api-tls\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.629657 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.629590    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx997\" (UniqueName: \"kubernetes.io/projected/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-kube-api-access-mx997\") pod \"maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:35.659953 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:35.659924    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:36.042298 ip-10-0-129-77 kubenswrapper[4837]: W1101 22:58:36.042272    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8f4047_d714_4652_8d8d_7cb2bf0202ce.slice/crio-7a44a6be53b2ffc6388434d7c939ebe422349b4d01d2f9b0ed77204b8958050e WatchSource:0}: Error finding container 7a44a6be53b2ffc6388434d7c939ebe422349b4d01d2f9b0ed77204b8958050e: Status 404 returned error can't find the container with id 7a44a6be53b2ffc6388434d7c939ebe422349b4d01d2f9b0ed77204b8958050e
Nov 01 22:58:36.042750 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:36.042729    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"]
Nov 01 22:58:36.136855 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:36.136811    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl" event={"ID":"ec8f4047-d714-4652-8d8d-7cb2bf0202ce","Type":"ContainerStarted","Data":"7a44a6be53b2ffc6388434d7c939ebe422349b4d01d2f9b0ed77204b8958050e"}
Nov 01 22:58:37.145628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:37.145543    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl" event={"ID":"ec8f4047-d714-4652-8d8d-7cb2bf0202ce","Type":"ContainerStarted","Data":"b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3"}
Nov 01 22:58:37.146008 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:37.145656    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:37.290560 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:37.290507    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl" podStartSLOduration=1.454517584 podStartE2EDuration="2.290493083s" totalImagesPullingTime="835.975499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:37.287986779 +0000 UTC m=+2098.877185841" watchObservedRunningTime="2026-11-01 22:58:37.290493083 +0000 UTC m=+2098.879692143"
Nov 01 22:58:38.836628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:38.836596    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:58:38.837983 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:38.837966    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 22:58:43.165488 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:58:43.165459    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 22:58:48.815226 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:58:48.815149    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:59:05.948889 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:05.948843    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 22:59:10.911917 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:10.911879    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"]
Nov 01 22:59:11.030329 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:11.030284    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"]
Nov 01 22:59:15.920504 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:15.920444    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl" podUID="a1715f8e-51f0-4d86-80b3-dc6e43ea14dd" containerName="payload-processing" containerID="cri-o://042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0" gracePeriod=25
Nov 01 22:59:16.031565 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.031524    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9" podUID="76032052-d23e-4753-9c1b-e3f59d504acf" containerName="payload-pre-processing" containerID="cri-o://ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce" gracePeriod=25
Nov 01 22:59:16.267094 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.267022    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:59:16.333443 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.333420    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:59:16.374939 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.374907    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-plugins-config-volume\") pod \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\" (UID: \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\") "
Nov 01 22:59:16.375111 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.374951    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-kube-api-access-2n8j5\" (UniqueName: \"kubernetes.io/projected/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-kube-api-access-2n8j5\") pod \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\" (UID: \"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd\") "
Nov 01 22:59:16.375381 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.375350    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-plugins-config-volume" pod "a1715f8e-51f0-4d86-80b3-dc6e43ea14dd" (UID: "a1715f8e-51f0-4d86-80b3-dc6e43ea14dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:59:16.375559 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.375542    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:16.377404 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.377375    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-kube-api-access-2n8j5" pod "a1715f8e-51f0-4d86-80b3-dc6e43ea14dd" (UID: "a1715f8e-51f0-4d86-80b3-dc6e43ea14dd"). InnerVolumeSpecName "kube-api-access-2n8j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:16.476450 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.476371    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76032052-d23e-4753-9c1b-e3f59d504acf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76032052-d23e-4753-9c1b-e3f59d504acf-plugins-config-volume\") pod \"76032052-d23e-4753-9c1b-e3f59d504acf\" (UID: \"76032052-d23e-4753-9c1b-e3f59d504acf\") "
Nov 01 22:59:16.476628 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.476469    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76032052-d23e-4753-9c1b-e3f59d504acf-kube-api-access-jmrn4\" (UniqueName: \"kubernetes.io/projected/76032052-d23e-4753-9c1b-e3f59d504acf-kube-api-access-jmrn4\") pod \"76032052-d23e-4753-9c1b-e3f59d504acf\" (UID: \"76032052-d23e-4753-9c1b-e3f59d504acf\") "
Nov 01 22:59:16.476868 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.476837    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76032052-d23e-4753-9c1b-e3f59d504acf-plugins-config-volume" pod "76032052-d23e-4753-9c1b-e3f59d504acf" (UID: "76032052-d23e-4753-9c1b-e3f59d504acf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:59:16.477246 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.477223    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n8j5\" (UniqueName: \"kubernetes.io/projected/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd-kube-api-access-2n8j5\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:16.477345 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.477249    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76032052-d23e-4753-9c1b-e3f59d504acf-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:16.478806 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.478784    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76032052-d23e-4753-9c1b-e3f59d504acf-kube-api-access-jmrn4" pod "76032052-d23e-4753-9c1b-e3f59d504acf" (UID: "76032052-d23e-4753-9c1b-e3f59d504acf"). InnerVolumeSpecName "kube-api-access-jmrn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:16.562338 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.562305    4837 generic.go:356] "Generic (PLEG): container finished" podID="a1715f8e-51f0-4d86-80b3-dc6e43ea14dd" containerID="042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0" exitCode=0
Nov 01 22:59:16.562495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.562361    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"
Nov 01 22:59:16.562495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.562394    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl" event={"ID":"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd","Type":"ContainerDied","Data":"042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0"}
Nov 01 22:59:16.562495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.562442    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl" event={"ID":"a1715f8e-51f0-4d86-80b3-dc6e43ea14dd","Type":"ContainerDied","Data":"53afa1668b168f26635b7fb7d9827b93fb78b100a74b86644cd1e16441842dca"}
Nov 01 22:59:16.562495 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.562473    4837 scope.go:122] "RemoveContainer" containerID="042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0"
Nov 01 22:59:16.564071 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.564020    4837 generic.go:356] "Generic (PLEG): container finished" podID="76032052-d23e-4753-9c1b-e3f59d504acf" containerID="ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce" exitCode=0
Nov 01 22:59:16.564176 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.564096    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"
Nov 01 22:59:16.564176 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.564120    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9" event={"ID":"76032052-d23e-4753-9c1b-e3f59d504acf","Type":"ContainerDied","Data":"ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce"}
Nov 01 22:59:16.564260 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.564182    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9" event={"ID":"76032052-d23e-4753-9c1b-e3f59d504acf","Type":"ContainerDied","Data":"48505e0574900db77b298d8c97e3ca552c96d780a63c547b6d1415bab7d65f86"}
Nov 01 22:59:16.573924 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.573909    4837 scope.go:122] "RemoveContainer" containerID="042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0"
Nov 01 22:59:16.574233 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:59:16.574216    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0\": container with ID starting with 042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0 not found: ID does not exist" containerID="042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0"
Nov 01 22:59:16.574291 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.574241    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0"} err="failed to get container status \"042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0\": rpc error: code = NotFound desc = could not find container \"042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0\": container with ID starting with 042f731f321eac84d39a64a449ae9c42b1e80da5a5f1577c27b13850f5bb11b0 not found: ID does not exist"
Nov 01 22:59:16.574291 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.574258    4837 scope.go:122] "RemoveContainer" containerID="ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce"
Nov 01 22:59:16.578927 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.578906    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrn4\" (UniqueName: \"kubernetes.io/projected/76032052-d23e-4753-9c1b-e3f59d504acf-kube-api-access-jmrn4\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:16.586479 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.586454    4837 scope.go:122] "RemoveContainer" containerID="ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce"
Nov 01 22:59:16.586745 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:59:16.586728    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce\": container with ID starting with ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce not found: ID does not exist" containerID="ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce"
Nov 01 22:59:16.586797 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.586754    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce"} err="failed to get container status \"ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce\": rpc error: code = NotFound desc = could not find container \"ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce\": container with ID starting with ec024446d095655d66e7c6fcf9207401c66e4d46e2c7cf22d05ff68ec4e8e0ce not found: ID does not exist"
Nov 01 22:59:16.718426 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.718378    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"]
Nov 01 22:59:16.732630 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.732557    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-xgzv9"]
Nov 01 22:59:16.825662 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.825628    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"]
Nov 01 22:59:16.829636 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:16.829612    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-945efd02-58b9cd96fc-thwzl"]
Nov 01 22:59:18.306406 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:18.306377    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"]
Nov 01 22:59:18.306802 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:18.306596    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" containerID="cri-o://cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66" gracePeriod=30
Nov 01 22:59:18.527644 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:18.527613    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76032052-d23e-4753-9c1b-e3f59d504acf" path="/var/lib/kubelet/pods/76032052-d23e-4753-9c1b-e3f59d504acf/volumes"
Nov 01 22:59:18.528107 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:18.528091    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1715f8e-51f0-4d86-80b3-dc6e43ea14dd" path="/var/lib/kubelet/pods/a1715f8e-51f0-4d86-80b3-dc6e43ea14dd/volumes"
Nov 01 22:59:18.814718 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:59:18.814684    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 22:59:20.008553 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:20.008519    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:59:20.008988 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:20.008578    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:59:22.477302 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:22.477267    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"]
Nov 01 22:59:23.591911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.591888    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:59:23.641600 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.641568    4837 generic.go:356] "Generic (PLEG): container finished" podID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerID="cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66" exitCode=0
Nov 01 22:59:23.641772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.641634    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"
Nov 01 22:59:23.641772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.641657    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" event={"ID":"6e90043e-2824-4385-9c8c-ae50bfa42a60","Type":"ContainerDied","Data":"cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66"}
Nov 01 22:59:23.641772 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.641755    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n" event={"ID":"6e90043e-2824-4385-9c8c-ae50bfa42a60","Type":"ContainerDied","Data":"adc19c732c1d7c2c5a2ab5332a650c6d52a51309d1bc88ea248e749b5627c89a"}
Nov 01 22:59:23.641937 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.641786    4837 scope.go:122] "RemoveContainer" containerID="cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66"
Nov 01 22:59:23.653004 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.652977    4837 scope.go:122] "RemoveContainer" containerID="cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66"
Nov 01 22:59:23.653298 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:59:23.653280    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66\": container with ID starting with cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66 not found: ID does not exist" containerID="cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66"
Nov 01 22:59:23.653354 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.653307    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66"} err="failed to get container status \"cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66\": rpc error: code = NotFound desc = could not find container \"cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66\": container with ID starting with cba7244e092385716eba3de399f6607b54a1dac8e2aeaead4fe85908bbddca66 not found: ID does not exist"
Nov 01 22:59:23.689204 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689178    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-credential-socket\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689298 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689275    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-socket\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689357 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689320    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-token\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-token\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689357 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689340    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-data\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689464 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689357    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-podinfo\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689464 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689413    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-credential-socket" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.689637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689467    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-certs\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689500    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-kube-api-access-r2kxs\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-kube-api-access-r2kxs\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689637 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689619    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-data" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.689779 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689633    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-socket" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.689779 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689667    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-envoy\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689779 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689732    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e90043e-2824-4385-9c8c-ae50bfa42a60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e90043e-2824-4385-9c8c-ae50bfa42a60-istiod-ca-cert\") pod \"6e90043e-2824-4385-9c8c-ae50bfa42a60\" (UID: \"6e90043e-2824-4385-9c8c-ae50bfa42a60\") "
Nov 01 22:59:23.689946 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.689731    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-certs" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.690080 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.690052    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e90043e-2824-4385-9c8c-ae50bfa42a60-istiod-ca-cert" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:59:23.690500 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.690480    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.690566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.690509    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.690566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.690526    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e90043e-2824-4385-9c8c-ae50bfa42a60-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.690566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.690541    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.690566 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.690555    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.691781 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.691754    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-kube-api-access-r2kxs" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "kube-api-access-r2kxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:23.692012 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.691984    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-token" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:23.692158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.692026    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-podinfo" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:59:23.692158 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.692067    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-envoy" pod "6e90043e-2824-4385-9c8c-ae50bfa42a60" (UID: "6e90043e-2824-4385-9c8c-ae50bfa42a60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.792380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.792342    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.792380 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.792381    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.792638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.792397    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e90043e-2824-4385-9c8c-ae50bfa42a60-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.792638 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:23.792413    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2kxs\" (UniqueName: \"kubernetes.io/projected/6e90043e-2824-4385-9c8c-ae50bfa42a60-kube-api-access-r2kxs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:24.058232 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:24.058201    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"]
Nov 01 22:59:24.061906 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:24.061878    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-bps8n"]
Nov 01 22:59:24.527672 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:24.527643    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" path="/var/lib/kubelet/pods/6e90043e-2824-4385-9c8c-ae50bfa42a60/volumes"
Nov 01 22:59:37.499893 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.499803    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerName="main" containerID="cri-o://83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd" gracePeriod=45
Nov 01 22:59:37.804056 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.804019    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:59:37.804799 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.804767    4837 generic.go:356] "Generic (PLEG): container finished" podID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerID="83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd" exitCode=0
Nov 01 22:59:37.804932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.804845    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" event={"ID":"e8ebd46d-560f-40e6-b31a-2946fc22c11d","Type":"ContainerDied","Data":"83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd"}
Nov 01 22:59:37.804932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.804895    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs" event={"ID":"e8ebd46d-560f-40e6-b31a-2946fc22c11d","Type":"ContainerDied","Data":"2198dcf82dc94f5b3da7515cb14e5d876df243fbfc033491d536cb7a256f7758"}
Nov 01 22:59:37.804932 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.804918    4837 scope.go:122] "RemoveContainer" containerID="83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd"
Nov 01 22:59:37.815544 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.815516    4837 scope.go:122] "RemoveContainer" containerID="83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd"
Nov 01 22:59:37.815834 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:59:37.815809    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd\": container with ID starting with 83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd not found: ID does not exist" containerID="83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd"
Nov 01 22:59:37.815916 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.815847    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd"} err="failed to get container status \"83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd\": rpc error: code = NotFound desc = could not find container \"83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd\": container with ID starting with 83aa54f44e42626a8d84e77829a2621f8aae79dc46ae35236bf95021041c0efd not found: ID does not exist"
Nov 01 22:59:37.976035 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976007    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-model-cache\") pod \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") "
Nov 01 22:59:37.976035 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976069    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-home\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-home\") pod \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") "
Nov 01 22:59:37.976312 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976110    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tls-certs\") pod \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") "
Nov 01 22:59:37.976312 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976151    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tmp-dir\") pod \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") "
Nov 01 22:59:37.976418 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976306    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-model-cache" pod "e8ebd46d-560f-40e6-b31a-2946fc22c11d" (UID: "e8ebd46d-560f-40e6-b31a-2946fc22c11d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:37.976474 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976444    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ebd46d-560f-40e6-b31a-2946fc22c11d-kube-api-access-7fhvv\" (UniqueName: \"kubernetes.io/projected/e8ebd46d-560f-40e6-b31a-2946fc22c11d-kube-api-access-7fhvv\") pod \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") "
Nov 01 22:59:37.976530 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976512    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tmp-dir" pod "e8ebd46d-560f-40e6-b31a-2946fc22c11d" (UID: "e8ebd46d-560f-40e6-b31a-2946fc22c11d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:37.976589 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976520    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-dshm\") pod \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\" (UID: \"e8ebd46d-560f-40e6-b31a-2946fc22c11d\") "
Nov 01 22:59:37.976911 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.976879    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-home" pod "e8ebd46d-560f-40e6-b31a-2946fc22c11d" (UID: "e8ebd46d-560f-40e6-b31a-2946fc22c11d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:37.977396 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.977373    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:37.977514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.977399    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:37.977514 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.977409    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:37.979173 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.979136    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tls-certs" pod "e8ebd46d-560f-40e6-b31a-2946fc22c11d" (UID: "e8ebd46d-560f-40e6-b31a-2946fc22c11d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:59:37.979356 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.979215    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ebd46d-560f-40e6-b31a-2946fc22c11d-kube-api-access-7fhvv" pod "e8ebd46d-560f-40e6-b31a-2946fc22c11d" (UID: "e8ebd46d-560f-40e6-b31a-2946fc22c11d"). InnerVolumeSpecName "kube-api-access-7fhvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:37.979356 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:37.979305    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-dshm" pod "e8ebd46d-560f-40e6-b31a-2946fc22c11d" (UID: "e8ebd46d-560f-40e6-b31a-2946fc22c11d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:38.078621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:38.078523    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fhvv\" (UniqueName: \"kubernetes.io/projected/e8ebd46d-560f-40e6-b31a-2946fc22c11d-kube-api-access-7fhvv\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:38.078621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:38.078557    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8ebd46d-560f-40e6-b31a-2946fc22c11d-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:38.078621 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:38.078571    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8ebd46d-560f-40e6-b31a-2946fc22c11d-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:38.813211 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:38.813136    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"
Nov 01 22:59:38.967959 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:38.967934    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"]
Nov 01 22:59:39.014362 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:39.014328    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-945efd02/e2e-lifecycle-model-945efd02-kserve-79c78f4949-snjzs"]
Nov 01 22:59:40.527333 ip-10-0-129-77 kubenswrapper[4837]: I1101 22:59:40.527299    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" path="/var/lib/kubelet/pods/e8ebd46d-560f-40e6-b31a-2946fc22c11d/volumes"
Nov 01 22:59:48.813746 ip-10-0-129-77 kubenswrapper[4837]: E1101 22:59:48.813707    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:00:00.208552 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.208502    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"]
Nov 01 23:00:00.209661 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209638    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerName="main"
Nov 01 23:00:00.209726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209664    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerName="main"
Nov 01 23:00:00.209726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209711    4837 cpu_manager.go:414] "containerMap: removing container" podUID="76032052-d23e-4753-9c1b-e3f59d504acf" containerName="payload-pre-processing"
Nov 01 23:00:00.209726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209719    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76032052-d23e-4753-9c1b-e3f59d504acf" containerName="payload-pre-processing"
Nov 01 23:00:00.209823 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209739    4837 cpu_manager.go:414] "containerMap: removing container" podUID="a1715f8e-51f0-4d86-80b3-dc6e43ea14dd" containerName="payload-processing"
Nov 01 23:00:00.209823 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209747    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1715f8e-51f0-4d86-80b3-dc6e43ea14dd" containerName="payload-processing"
Nov 01 23:00:00.209823 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209770    4837 cpu_manager.go:414] "containerMap: removing container" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy"
Nov 01 23:00:00.209823 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209778    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy"
Nov 01 23:00:00.209955 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209942    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1715f8e-51f0-4d86-80b3-dc6e43ea14dd" containerName="payload-processing"
Nov 01 23:00:00.209989 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209960    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e90043e-2824-4385-9c8c-ae50bfa42a60" containerName="istio-proxy"
Nov 01 23:00:00.209989 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209980    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ebd46d-560f-40e6-b31a-2946fc22c11d" containerName="main"
Nov 01 23:00:00.210098 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.209995    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="76032052-d23e-4753-9c1b-e3f59d504acf" containerName="payload-pre-processing"
Nov 01 23:00:00.214347 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.214327    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"]
Nov 01 23:00:00.214452 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.214432    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:00.219009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.218992    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:00.240962 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.240933    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"]
Nov 01 23:00:00.244925 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.244911    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:00.255274 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.255253    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"]
Nov 01 23:00:00.259336 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.259321    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:00.270617 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.270595    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9665q\" (UniqueName: \"kubernetes.io/projected/d6af8618-cbd0-4678-9dfc-088e8337ed08-kube-api-access-9665q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn\" (UID: \"d6af8618-cbd0-4678-9dfc-088e8337ed08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:00.270715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.270698    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mhg\" (UniqueName: \"kubernetes.io/projected/5a841a30-8ea9-449e-bda1-1309b98b5141-kube-api-access-58mhg\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn\" (UID: \"5a841a30-8ea9-449e-bda1-1309b98b5141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:00.270774 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.270736    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a841a30-8ea9-449e-bda1-1309b98b5141-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn\" (UID: \"5a841a30-8ea9-449e-bda1-1309b98b5141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:00.270830 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.270822    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6af8618-cbd0-4678-9dfc-088e8337ed08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn\" (UID: \"d6af8618-cbd0-4678-9dfc-088e8337ed08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:00.273185 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.273166    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:00:00.288021 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.288000    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"]
Nov 01 23:00:00.303191 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.303167    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"]
Nov 01 23:00:00.308875 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.308851    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"]
Nov 01 23:00:00.314379 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.314364    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:00.323534 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.323511    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"]
Nov 01 23:00:00.329364 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.329344    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:00.340208 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.338185    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"]
Nov 01 23:00:00.343172 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.343147    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"]
Nov 01 23:00:00.357254 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.357200    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"]
Nov 01 23:00:00.365115 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.365094    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:00.366210 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.366188    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"]
Nov 01 23:00:00.372229 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372199    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt48s\" (UniqueName: \"kubernetes.io/projected/75034634-c95f-4cb2-aa56-897dc81cef3f-kube-api-access-pt48s\") pod \"maas-api-key-cleanup-29892900-sfj9q\" (UID: \"75034634-c95f-4cb2-aa56-897dc81cef3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:00.372377 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372257    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75034634-c95f-4cb2-aa56-897dc81cef3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-sfj9q\" (UID: \"75034634-c95f-4cb2-aa56-897dc81cef3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:00.372377 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372302    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd9a7a-987c-47b2-9138-49889b712d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb\" (UID: \"fabd9a7a-987c-47b2-9138-49889b712d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:00.372377 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372329    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz794\" (UniqueName: \"kubernetes.io/projected/8c17bfba-c48b-41de-872a-c491be361d55-kube-api-access-nz794\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9\" (UID: \"8c17bfba-c48b-41de-872a-c491be361d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:00.372547 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372421    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9m5\" (UniqueName: \"kubernetes.io/projected/ea5787d4-6244-4b0f-a2ac-84044b041586-kube-api-access-dn9m5\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh\" (UID: \"ea5787d4-6244-4b0f-a2ac-84044b041586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:00.372547 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372520    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9665q\" (UniqueName: \"kubernetes.io/projected/d6af8618-cbd0-4678-9dfc-088e8337ed08-kube-api-access-9665q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn\" (UID: \"d6af8618-cbd0-4678-9dfc-088e8337ed08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:00.372658 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372564    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5787d4-6244-4b0f-a2ac-84044b041586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh\" (UID: \"ea5787d4-6244-4b0f-a2ac-84044b041586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:00.372658 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372644    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mhg\" (UniqueName: \"kubernetes.io/projected/5a841a30-8ea9-449e-bda1-1309b98b5141-kube-api-access-58mhg\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn\" (UID: \"5a841a30-8ea9-449e-bda1-1309b98b5141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:00.372760 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372690    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a841a30-8ea9-449e-bda1-1309b98b5141-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn\" (UID: \"5a841a30-8ea9-449e-bda1-1309b98b5141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:00.372760 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372711    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:00.372881 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.372760    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6af8618-cbd0-4678-9dfc-088e8337ed08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn\" (UID: \"d6af8618-cbd0-4678-9dfc-088e8337ed08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:00.373608 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.373586    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r9zb\" (UniqueName: \"kubernetes.io/projected/fabd9a7a-987c-47b2-9138-49889b712d21-kube-api-access-4r9zb\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb\" (UID: \"fabd9a7a-987c-47b2-9138-49889b712d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:00.373733 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.373696    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c17bfba-c48b-41de-872a-c491be361d55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9\" (UID: \"8c17bfba-c48b-41de-872a-c491be361d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:00.373807 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.373780    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6af8618-cbd0-4678-9dfc-088e8337ed08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn\" (UID: \"d6af8618-cbd0-4678-9dfc-088e8337ed08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:00.374407 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.374378    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a841a30-8ea9-449e-bda1-1309b98b5141-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn\" (UID: \"5a841a30-8ea9-449e-bda1-1309b98b5141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:00.375259 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.375238    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"]
Nov 01 23:00:00.390416 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.390393    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"]
Nov 01 23:00:00.394891 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.394872    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:00.425374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.425346    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"]
Nov 01 23:00:00.437626 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.437600    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"]
Nov 01 23:00:00.441609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.441578    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9665q\" (UniqueName: \"kubernetes.io/projected/d6af8618-cbd0-4678-9dfc-088e8337ed08-kube-api-access-9665q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn\" (UID: \"d6af8618-cbd0-4678-9dfc-088e8337ed08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:00.450897 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.450868    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"]
Nov 01 23:00:00.457659 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.457631    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"]
Nov 01 23:00:00.457784 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.457745    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:00.458101 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.458028    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mhg\" (UniqueName: \"kubernetes.io/projected/5a841a30-8ea9-449e-bda1-1309b98b5141-kube-api-access-58mhg\") pod \"maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn\" (UID: \"5a841a30-8ea9-449e-bda1-1309b98b5141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:00.480127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.469299    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"]
Nov 01 23:00:00.484823 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.484797    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5787d4-6244-4b0f-a2ac-84044b041586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh\" (UID: \"ea5787d4-6244-4b0f-a2ac-84044b041586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:00.484924 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.484875    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbs9\" (UniqueName: \"kubernetes.io/projected/f8d859c5-be57-4861-b33f-ecc6cb4d4178-kube-api-access-vtbs9\") pod \"maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf\" (UID: \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:00.484976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.484938    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d859c5-be57-4861-b33f-ecc6cb4d4178-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf\" (UID: \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:00.485143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485121    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r9zb\" (UniqueName: \"kubernetes.io/projected/fabd9a7a-987c-47b2-9138-49889b712d21-kube-api-access-4r9zb\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb\" (UID: \"fabd9a7a-987c-47b2-9138-49889b712d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:00.485193 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485159    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbg4\" (UniqueName: \"kubernetes.io/projected/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-kube-api-access-2nbg4\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp\" (UID: \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:00.485237 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485220    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c17bfba-c48b-41de-872a-c491be361d55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9\" (UID: \"8c17bfba-c48b-41de-872a-c491be361d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:00.485284 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485247    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f78edcf-5daf-47ba-85f0-050cf11657fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w\" (UID: \"3f78edcf-5daf-47ba-85f0-050cf11657fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:00.485446 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485428    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzxlt\" (UniqueName: \"kubernetes.io/projected/3f78edcf-5daf-47ba-85f0-050cf11657fb-kube-api-access-nzxlt\") pod \"maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w\" (UID: \"3f78edcf-5daf-47ba-85f0-050cf11657fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:00.485525 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485472    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp\" (UID: \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:00.485616 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485601    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt48s\" (UniqueName: \"kubernetes.io/projected/75034634-c95f-4cb2-aa56-897dc81cef3f-kube-api-access-pt48s\") pod \"maas-api-key-cleanup-29892900-sfj9q\" (UID: \"75034634-c95f-4cb2-aa56-897dc81cef3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:00.485677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485625    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5787d4-6244-4b0f-a2ac-84044b041586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh\" (UID: \"ea5787d4-6244-4b0f-a2ac-84044b041586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:00.485677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485643    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75034634-c95f-4cb2-aa56-897dc81cef3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-sfj9q\" (UID: \"75034634-c95f-4cb2-aa56-897dc81cef3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:00.485783 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485689    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd9a7a-987c-47b2-9138-49889b712d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb\" (UID: \"fabd9a7a-987c-47b2-9138-49889b712d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:00.485783 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485718    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz794\" (UniqueName: \"kubernetes.io/projected/8c17bfba-c48b-41de-872a-c491be361d55-kube-api-access-nz794\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9\" (UID: \"8c17bfba-c48b-41de-872a-c491be361d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:00.485892 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485789    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9m5\" (UniqueName: \"kubernetes.io/projected/ea5787d4-6244-4b0f-a2ac-84044b041586-kube-api-access-dn9m5\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh\" (UID: \"ea5787d4-6244-4b0f-a2ac-84044b041586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:00.485990 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.485971    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c17bfba-c48b-41de-872a-c491be361d55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9\" (UID: \"8c17bfba-c48b-41de-872a-c491be361d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:00.486404 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.486385    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75034634-c95f-4cb2-aa56-897dc81cef3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-sfj9q\" (UID: \"75034634-c95f-4cb2-aa56-897dc81cef3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:00.486452 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.486435    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd9a7a-987c-47b2-9138-49889b712d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb\" (UID: \"fabd9a7a-987c-47b2-9138-49889b712d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:00.515560 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.515424    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"]
Nov 01 23:00:00.529240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.529210    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:00.536674 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.536650    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:00.541747 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.541718    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r9zb\" (UniqueName: \"kubernetes.io/projected/fabd9a7a-987c-47b2-9138-49889b712d21-kube-api-access-4r9zb\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb\" (UID: \"fabd9a7a-987c-47b2-9138-49889b712d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:00.543976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.543789    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9m5\" (UniqueName: \"kubernetes.io/projected/ea5787d4-6244-4b0f-a2ac-84044b041586-kube-api-access-dn9m5\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh\" (UID: \"ea5787d4-6244-4b0f-a2ac-84044b041586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:00.544921 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.544897    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt48s\" (UniqueName: \"kubernetes.io/projected/75034634-c95f-4cb2-aa56-897dc81cef3f-kube-api-access-pt48s\") pod \"maas-api-key-cleanup-29892900-sfj9q\" (UID: \"75034634-c95f-4cb2-aa56-897dc81cef3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:00.544998 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.544972    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz794\" (UniqueName: \"kubernetes.io/projected/8c17bfba-c48b-41de-872a-c491be361d55-kube-api-access-nz794\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9\" (UID: \"8c17bfba-c48b-41de-872a-c491be361d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:00.560693 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.560667    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:00.576074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.576018    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.587137    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d859c5-be57-4861-b33f-ecc6cb4d4178-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf\" (UID: \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.587254    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0172cd03-fd27-4b92-920b-ba2ee1b89f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd\" (UID: \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.587306    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbg4\" (UniqueName: \"kubernetes.io/projected/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-kube-api-access-2nbg4\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp\" (UID: \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.587364    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f78edcf-5daf-47ba-85f0-050cf11657fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w\" (UID: \"3f78edcf-5daf-47ba-85f0-050cf11657fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.587414    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzxlt\" (UniqueName: \"kubernetes.io/projected/3f78edcf-5daf-47ba-85f0-050cf11657fb-kube-api-access-nzxlt\") pod \"maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w\" (UID: \"3f78edcf-5daf-47ba-85f0-050cf11657fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.587448    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp\" (UID: \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.587561    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn8dl\" (UniqueName: \"kubernetes.io/projected/0172cd03-fd27-4b92-920b-ba2ee1b89f90-kube-api-access-gn8dl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd\" (UID: \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.587749    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbs9\" (UniqueName: \"kubernetes.io/projected/f8d859c5-be57-4861-b33f-ecc6cb4d4178-kube-api-access-vtbs9\") pod \"maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf\" (UID: \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.588775    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d859c5-be57-4861-b33f-ecc6cb4d4178-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf\" (UID: \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.589716    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp\" (UID: \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:00.592087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.590327    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f78edcf-5daf-47ba-85f0-050cf11657fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w\" (UID: \"3f78edcf-5daf-47ba-85f0-050cf11657fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:00.639227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.629470    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:00.656999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.655216    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:00.661300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.661254    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbg4\" (UniqueName: \"kubernetes.io/projected/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-kube-api-access-2nbg4\") pod \"maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp\" (UID: \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:00.662312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.662288    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbs9\" (UniqueName: \"kubernetes.io/projected/f8d859c5-be57-4861-b33f-ecc6cb4d4178-kube-api-access-vtbs9\") pod \"maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf\" (UID: \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:00.663795 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.663738    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzxlt\" (UniqueName: \"kubernetes.io/projected/3f78edcf-5daf-47ba-85f0-050cf11657fb-kube-api-access-nzxlt\") pod \"maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w\" (UID: \"3f78edcf-5daf-47ba-85f0-050cf11657fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:00.692511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.690438    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn8dl\" (UniqueName: \"kubernetes.io/projected/0172cd03-fd27-4b92-920b-ba2ee1b89f90-kube-api-access-gn8dl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd\" (UID: \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:00.692511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.690629    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0172cd03-fd27-4b92-920b-ba2ee1b89f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd\" (UID: \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:00.692511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.691964    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:00.693863 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.693805    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0172cd03-fd27-4b92-920b-ba2ee1b89f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd\" (UID: \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:00.702067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.699667    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:00.710670 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.710484    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:00.759293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.759253    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn8dl\" (UniqueName: \"kubernetes.io/projected/0172cd03-fd27-4b92-920b-ba2ee1b89f90-kube-api-access-gn8dl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd\" (UID: \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:00.831666 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.803351    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:00.945449 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.944606    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"]
Nov 01 23:00:00.950522 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:00.950493    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6af8618_cbd0_4678_9dfc_088e8337ed08.slice/crio-dc083da79831d63b3945eb1f093155faa8d543e7a8921fd6fab627b1483135b4 WatchSource:0}: Error finding container dc083da79831d63b3945eb1f093155faa8d543e7a8921fd6fab627b1483135b4: Status 404 returned error can't find the container with id dc083da79831d63b3945eb1f093155faa8d543e7a8921fd6fab627b1483135b4
Nov 01 23:00:00.973666 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:00.973643    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"]
Nov 01 23:00:00.977332 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:00.977294    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a841a30_8ea9_449e_bda1_1309b98b5141.slice/crio-428ed9d663eb8ff885ec1f966732f3109be9f6649ab501bf9eaaece0fee6ee33 WatchSource:0}: Error finding container 428ed9d663eb8ff885ec1f966732f3109be9f6649ab501bf9eaaece0fee6ee33: Status 404 returned error can't find the container with id 428ed9d663eb8ff885ec1f966732f3109be9f6649ab501bf9eaaece0fee6ee33
Nov 01 23:00:01.010447 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.010415    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" event={"ID":"5a841a30-8ea9-449e-bda1-1309b98b5141","Type":"ContainerStarted","Data":"428ed9d663eb8ff885ec1f966732f3109be9f6649ab501bf9eaaece0fee6ee33"}
Nov 01 23:00:01.012093 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.012025    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" event={"ID":"d6af8618-cbd0-4678-9dfc-088e8337ed08","Type":"ContainerStarted","Data":"dc083da79831d63b3945eb1f093155faa8d543e7a8921fd6fab627b1483135b4"}
Nov 01 23:00:01.021787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.021735    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"]
Nov 01 23:00:01.021979 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:01.021954    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabd9a7a_987c_47b2_9138_49889b712d21.slice/crio-df800499b4c7150041bd26309d15d05aee22df27ad744d9b222faa6605c879af WatchSource:0}: Error finding container df800499b4c7150041bd26309d15d05aee22df27ad744d9b222faa6605c879af: Status 404 returned error can't find the container with id df800499b4c7150041bd26309d15d05aee22df27ad744d9b222faa6605c879af
Nov 01 23:00:01.191199 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:01.191166    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5787d4_6244_4b0f_a2ac_84044b041586.slice/crio-3a24352a7c82259db4511fcfd73590b62afc5a46d3c2e7afb69af768ae7a3e5a WatchSource:0}: Error finding container 3a24352a7c82259db4511fcfd73590b62afc5a46d3c2e7afb69af768ae7a3e5a: Status 404 returned error can't find the container with id 3a24352a7c82259db4511fcfd73590b62afc5a46d3c2e7afb69af768ae7a3e5a
Nov 01 23:00:01.192395 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.192370    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"]
Nov 01 23:00:01.259368 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.259338    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"]
Nov 01 23:00:01.290454 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.290425    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"]
Nov 01 23:00:01.317409 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.317386    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"]
Nov 01 23:00:01.346176 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.346153    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"]
Nov 01 23:00:01.348266 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:01.348007    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d89a4be_f89e_4f7a_8c7f_5b4e18bb09c5.slice/crio-6f22cfb8e3512d9a919e008aa52bd74c9855fee1c0008332972aff26353b7fad WatchSource:0}: Error finding container 6f22cfb8e3512d9a919e008aa52bd74c9855fee1c0008332972aff26353b7fad: Status 404 returned error can't find the container with id 6f22cfb8e3512d9a919e008aa52bd74c9855fee1c0008332972aff26353b7fad
Nov 01 23:00:01.351096 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.351071    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"]
Nov 01 23:00:01.386101 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:01.386066    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"]
Nov 01 23:00:01.386531 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:01.386505    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d859c5_be57_4861_b33f_ecc6cb4d4178.slice/crio-3a59c120cec808dbc243d71987df84239e61c2027293e94fd766f32a486f6c7b WatchSource:0}: Error finding container 3a59c120cec808dbc243d71987df84239e61c2027293e94fd766f32a486f6c7b: Status 404 returned error can't find the container with id 3a59c120cec808dbc243d71987df84239e61c2027293e94fd766f32a486f6c7b
Nov 01 23:00:01.432195 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:01.432011    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f78edcf_5daf_47ba_85f0_050cf11657fb.slice/crio-c6028276d5d5c42a0787eb609978855f91de84e8f5f22c5c67dc1ec3ea6cf4a3 WatchSource:0}: Error finding container c6028276d5d5c42a0787eb609978855f91de84e8f5f22c5c67dc1ec3ea6cf4a3: Status 404 returned error can't find the container with id c6028276d5d5c42a0787eb609978855f91de84e8f5f22c5c67dc1ec3ea6cf4a3
Nov 01 23:00:01.432558 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:01.432540    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75034634_c95f_4cb2_aa56_897dc81cef3f.slice/crio-0e8af44db2628afceefac4605ae5fe6863a757f2ccdb4271f45888052e369442 WatchSource:0}: Error finding container 0e8af44db2628afceefac4605ae5fe6863a757f2ccdb4271f45888052e369442: Status 404 returned error can't find the container with id 0e8af44db2628afceefac4605ae5fe6863a757f2ccdb4271f45888052e369442
Nov 01 23:00:02.021488 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.021456    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" event={"ID":"75034634-c95f-4cb2-aa56-897dc81cef3f","Type":"ContainerStarted","Data":"0e8af44db2628afceefac4605ae5fe6863a757f2ccdb4271f45888052e369442"}
Nov 01 23:00:02.022779 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.022749    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" event={"ID":"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5","Type":"ContainerStarted","Data":"6f22cfb8e3512d9a919e008aa52bd74c9855fee1c0008332972aff26353b7fad"}
Nov 01 23:00:02.024401 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.024378    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" event={"ID":"ea5787d4-6244-4b0f-a2ac-84044b041586","Type":"ContainerStarted","Data":"5c26e816a32a3917fa345921c44c99ea7c3d2541017685d774e78a63b1d66921"}
Nov 01 23:00:02.024524 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.024434    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" event={"ID":"ea5787d4-6244-4b0f-a2ac-84044b041586","Type":"ContainerStarted","Data":"3a24352a7c82259db4511fcfd73590b62afc5a46d3c2e7afb69af768ae7a3e5a"}
Nov 01 23:00:02.025674 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.025652    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" event={"ID":"8c17bfba-c48b-41de-872a-c491be361d55","Type":"ContainerStarted","Data":"7042e9049b132e9e1f48413085adb8fa67a5e0fac3df3a85a0fbb27bd37006b1"}
Nov 01 23:00:02.026876 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.026852    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w" event={"ID":"3f78edcf-5daf-47ba-85f0-050cf11657fb","Type":"ContainerStarted","Data":"c6028276d5d5c42a0787eb609978855f91de84e8f5f22c5c67dc1ec3ea6cf4a3"}
Nov 01 23:00:02.028425 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.028401    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" event={"ID":"5a841a30-8ea9-449e-bda1-1309b98b5141","Type":"ContainerStarted","Data":"11500f6f21540074dac3c02891996fa386bc621b3e2a423f8c71a7313a5e4ff6"}
Nov 01 23:00:02.030013 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.029989    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" event={"ID":"fabd9a7a-987c-47b2-9138-49889b712d21","Type":"ContainerStarted","Data":"3ee6c4b3c309fe235573f85525ed27bb04606f6c6810175195d1eab7f4ef8f5a"}
Nov 01 23:00:02.030132 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.030036    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" event={"ID":"fabd9a7a-987c-47b2-9138-49889b712d21","Type":"ContainerStarted","Data":"df800499b4c7150041bd26309d15d05aee22df27ad744d9b222faa6605c879af"}
Nov 01 23:00:02.032355 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.032332    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" event={"ID":"d6af8618-cbd0-4678-9dfc-088e8337ed08","Type":"ContainerStarted","Data":"e86c13d6233c804c8b8df8b15ac3b7a635671b8f1017a6799f400b49fa5c7f58"}
Nov 01 23:00:02.035446 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.035247    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" event={"ID":"0172cd03-fd27-4b92-920b-ba2ee1b89f90","Type":"ContainerStarted","Data":"15fc2801ac7521c232e6065204535ad0b65e7ed2b253330c99d43c61f454b8ba"}
Nov 01 23:00:02.036817 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.036799    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf" event={"ID":"f8d859c5-be57-4861-b33f-ecc6cb4d4178","Type":"ContainerStarted","Data":"3a59c120cec808dbc243d71987df84239e61c2027293e94fd766f32a486f6c7b"}
Nov 01 23:00:02.138299 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.138245    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" podStartSLOduration=2.138231197 podStartE2EDuration="2.138231197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.131452814 +0000 UTC m=+2183.720651875" watchObservedRunningTime="2026-11-01 23:00:02.138231197 +0000 UTC m=+2183.727430258"
Nov 01 23:00:02.247762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.247699    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" podStartSLOduration=2.247682358 podStartE2EDuration="2.247682358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.238847954 +0000 UTC m=+2183.828047016" watchObservedRunningTime="2026-11-01 23:00:02.247682358 +0000 UTC m=+2183.836881422"
Nov 01 23:00:02.348262 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.347321    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" podStartSLOduration=2.347307891 podStartE2EDuration="2.347307891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.345371666 +0000 UTC m=+2183.934570727" watchObservedRunningTime="2026-11-01 23:00:02.347307891 +0000 UTC m=+2183.936506961"
Nov 01 23:00:02.461481 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:02.461435    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" podStartSLOduration=2.461421363 podStartE2EDuration="2.461421363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.460291054 +0000 UTC m=+2184.049490114" watchObservedRunningTime="2026-11-01 23:00:02.461421363 +0000 UTC m=+2184.050620438"
Nov 01 23:00:03.047133 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.047105    4837 generic.go:356] "Generic (PLEG): container finished" podID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerID="5c26e816a32a3917fa345921c44c99ea7c3d2541017685d774e78a63b1d66921" exitCode=60
Nov 01 23:00:03.047238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.047187    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" event={"ID":"ea5787d4-6244-4b0f-a2ac-84044b041586","Type":"ContainerDied","Data":"5c26e816a32a3917fa345921c44c99ea7c3d2541017685d774e78a63b1d66921"}
Nov 01 23:00:03.047466 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.047440    4837 scope.go:122] "RemoveContainer" containerID="5c26e816a32a3917fa345921c44c99ea7c3d2541017685d774e78a63b1d66921"
Nov 01 23:00:03.048997 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.048970    4837 generic.go:356] "Generic (PLEG): container finished" podID="8c17bfba-c48b-41de-872a-c491be361d55" containerID="f31e33ed40b2983cc1c670a214d079707b4665a0e38853ec4535c8abfd55009b" exitCode=60
Nov 01 23:00:03.049127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.049085    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" event={"ID":"8c17bfba-c48b-41de-872a-c491be361d55","Type":"ContainerDied","Data":"f31e33ed40b2983cc1c670a214d079707b4665a0e38853ec4535c8abfd55009b"}
Nov 01 23:00:03.049353 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.049341    4837 scope.go:122] "RemoveContainer" containerID="f31e33ed40b2983cc1c670a214d079707b4665a0e38853ec4535c8abfd55009b"
Nov 01 23:00:03.051116 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.050986    4837 generic.go:356] "Generic (PLEG): container finished" podID="3f78edcf-5daf-47ba-85f0-050cf11657fb" containerID="54c71d19faba770f4656c0b40d74016866fc6370951b4c1f731cf4ab5ca2e725" exitCode=60
Nov 01 23:00:03.051179 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.051163    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w" event={"ID":"3f78edcf-5daf-47ba-85f0-050cf11657fb","Type":"ContainerDied","Data":"54c71d19faba770f4656c0b40d74016866fc6370951b4c1f731cf4ab5ca2e725"}
Nov 01 23:00:03.051283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.051267    4837 scope.go:122] "RemoveContainer" containerID="54c71d19faba770f4656c0b40d74016866fc6370951b4c1f731cf4ab5ca2e725"
Nov 01 23:00:03.052964 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.052946    4837 generic.go:356] "Generic (PLEG): container finished" podID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerID="11500f6f21540074dac3c02891996fa386bc621b3e2a423f8c71a7313a5e4ff6" exitCode=60
Nov 01 23:00:03.053023 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.053001    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" event={"ID":"5a841a30-8ea9-449e-bda1-1309b98b5141","Type":"ContainerDied","Data":"11500f6f21540074dac3c02891996fa386bc621b3e2a423f8c71a7313a5e4ff6"}
Nov 01 23:00:03.053292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.053274    4837 scope.go:122] "RemoveContainer" containerID="11500f6f21540074dac3c02891996fa386bc621b3e2a423f8c71a7313a5e4ff6"
Nov 01 23:00:03.054725 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.054708    4837 generic.go:356] "Generic (PLEG): container finished" podID="fabd9a7a-987c-47b2-9138-49889b712d21" containerID="3ee6c4b3c309fe235573f85525ed27bb04606f6c6810175195d1eab7f4ef8f5a" exitCode=60
Nov 01 23:00:03.054803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.054731    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" event={"ID":"fabd9a7a-987c-47b2-9138-49889b712d21","Type":"ContainerDied","Data":"3ee6c4b3c309fe235573f85525ed27bb04606f6c6810175195d1eab7f4ef8f5a"}
Nov 01 23:00:03.055013 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.054996    4837 scope.go:122] "RemoveContainer" containerID="3ee6c4b3c309fe235573f85525ed27bb04606f6c6810175195d1eab7f4ef8f5a"
Nov 01 23:00:03.056391 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.056371    4837 generic.go:356] "Generic (PLEG): container finished" podID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerID="e86c13d6233c804c8b8df8b15ac3b7a635671b8f1017a6799f400b49fa5c7f58" exitCode=60
Nov 01 23:00:03.056469 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.056436    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" event={"ID":"d6af8618-cbd0-4678-9dfc-088e8337ed08","Type":"ContainerDied","Data":"e86c13d6233c804c8b8df8b15ac3b7a635671b8f1017a6799f400b49fa5c7f58"}
Nov 01 23:00:03.056675 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.056656    4837 scope.go:122] "RemoveContainer" containerID="e86c13d6233c804c8b8df8b15ac3b7a635671b8f1017a6799f400b49fa5c7f58"
Nov 01 23:00:03.058099 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.058076    4837 generic.go:356] "Generic (PLEG): container finished" podID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerID="d3387a252e033eb20aadf30de9a97d9793449de42817e666fb81f14cbaf989f3" exitCode=60
Nov 01 23:00:03.058196 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.058166    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" event={"ID":"0172cd03-fd27-4b92-920b-ba2ee1b89f90","Type":"ContainerDied","Data":"d3387a252e033eb20aadf30de9a97d9793449de42817e666fb81f14cbaf989f3"}
Nov 01 23:00:03.058334 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.058319    4837 scope.go:122] "RemoveContainer" containerID="d3387a252e033eb20aadf30de9a97d9793449de42817e666fb81f14cbaf989f3"
Nov 01 23:00:03.060600 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.060572    4837 generic.go:356] "Generic (PLEG): container finished" podID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" containerID="56803d2d0e85f0415e4fa1e871a8c9fb19a26534910630c47ff18fea72f37e89" exitCode=60
Nov 01 23:00:03.060712 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.060691    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf" event={"ID":"f8d859c5-be57-4861-b33f-ecc6cb4d4178","Type":"ContainerDied","Data":"56803d2d0e85f0415e4fa1e871a8c9fb19a26534910630c47ff18fea72f37e89"}
Nov 01 23:00:03.060893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.060878    4837 scope.go:122] "RemoveContainer" containerID="56803d2d0e85f0415e4fa1e871a8c9fb19a26534910630c47ff18fea72f37e89"
Nov 01 23:00:03.062632 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.062612    4837 generic.go:356] "Generic (PLEG): container finished" podID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerID="4c5f8594ad874bb2f0b3a56336efec8fa12b5d05b9b216c6d65d3ac04025812f" exitCode=60
Nov 01 23:00:03.062733 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.062639    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" event={"ID":"75034634-c95f-4cb2-aa56-897dc81cef3f","Type":"ContainerDied","Data":"4c5f8594ad874bb2f0b3a56336efec8fa12b5d05b9b216c6d65d3ac04025812f"}
Nov 01 23:00:03.062888 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.062872    4837 scope.go:122] "RemoveContainer" containerID="4c5f8594ad874bb2f0b3a56336efec8fa12b5d05b9b216c6d65d3ac04025812f"
Nov 01 23:00:03.064506 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.064487    4837 generic.go:356] "Generic (PLEG): container finished" podID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerID="9037f37c02b06f2731a7d9f2073b65fa761d21bd5f08d5199ad10e5cbbf4fc8a" exitCode=60
Nov 01 23:00:03.064592 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.064511    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" event={"ID":"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5","Type":"ContainerDied","Data":"9037f37c02b06f2731a7d9f2073b65fa761d21bd5f08d5199ad10e5cbbf4fc8a"}
Nov 01 23:00:03.064764 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:03.064746    4837 scope.go:122] "RemoveContainer" containerID="9037f37c02b06f2731a7d9f2073b65fa761d21bd5f08d5199ad10e5cbbf4fc8a"
Nov 01 23:00:04.075974 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.075939    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" event={"ID":"d6af8618-cbd0-4678-9dfc-088e8337ed08","Type":"ContainerStarted","Data":"c56925acecdaa70b71e629baa93baab7018fbf1b17ea7e18ab42cfc9446cdaf8"}
Nov 01 23:00:04.081356 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.081328    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" event={"ID":"0172cd03-fd27-4b92-920b-ba2ee1b89f90","Type":"ContainerStarted","Data":"c0d42afd6d4ba7ab049b5d1396a3f1076f92901eafc2a989d5be79f46392d617"}
Nov 01 23:00:04.084414 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.084371    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf" event={"ID":"f8d859c5-be57-4861-b33f-ecc6cb4d4178","Type":"ContainerStarted","Data":"a654c38ced2382e744d28b47973a79c78ca7c3364920b0683d8f6287652e1c04"}
Nov 01 23:00:04.086491 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.086465    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" event={"ID":"75034634-c95f-4cb2-aa56-897dc81cef3f","Type":"ContainerStarted","Data":"b18d11d748898966b3a6be298a951922daf0d76245acd96ec34c193158201ec9"}
Nov 01 23:00:04.088481 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.088452    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" event={"ID":"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5","Type":"ContainerStarted","Data":"cba1f730d707e8366643ffe2ce0e508cfd89c7b47eb8c8ee3fa3e8ea4da80265"}
Nov 01 23:00:04.090221 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.090199    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" event={"ID":"ea5787d4-6244-4b0f-a2ac-84044b041586","Type":"ContainerStarted","Data":"7bbf60c35ef66ffea54197be6e48769751c251f624c7f2afe52f6019550e6578"}
Nov 01 23:00:04.091937 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.091916    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" event={"ID":"8c17bfba-c48b-41de-872a-c491be361d55","Type":"ContainerStarted","Data":"50268675d0e8253b92724ad796affac4cdbe8d1c3e30414fbdd57e85f9cf17ad"}
Nov 01 23:00:04.093729 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.093708    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w" event={"ID":"3f78edcf-5daf-47ba-85f0-050cf11657fb","Type":"ContainerStarted","Data":"964134efec7dbfecec914489139577f2c4d612b2273dcacd40a605abb9e9db57"}
Nov 01 23:00:04.095455 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.095437    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" event={"ID":"5a841a30-8ea9-449e-bda1-1309b98b5141","Type":"ContainerStarted","Data":"f12c817eda7e7dafb08cc692246012554965b37b4c0c64e4b49e415b631c4a7c"}
Nov 01 23:00:04.097235 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.097216    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" event={"ID":"fabd9a7a-987c-47b2-9138-49889b712d21","Type":"ContainerStarted","Data":"8e475cf166b9cd9d198aec67d4e38d315192c5e77f45f41fb14553176dc2eae1"}
Nov 01 23:00:04.255675 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.255617    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" podStartSLOduration=4.255600572 podStartE2EDuration="4.255600572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.251652415 +0000 UTC m=+2185.840851693" watchObservedRunningTime="2026-11-01 23:00:04.255600572 +0000 UTC m=+2185.844799634"
Nov 01 23:00:04.578263 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.578220    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" podStartSLOduration=4.578206493 podStartE2EDuration="4.578206493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.574539528 +0000 UTC m=+2186.163738589" watchObservedRunningTime="2026-11-01 23:00:04.578206493 +0000 UTC m=+2186.167405554"
Nov 01 23:00:04.686692 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.686642    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w" podStartSLOduration=4.686629381 podStartE2EDuration="4.686629381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.685242567 +0000 UTC m=+2186.274441627" watchObservedRunningTime="2026-11-01 23:00:04.686629381 +0000 UTC m=+2186.275828441"
Nov 01 23:00:04.795489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:04.795437    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" podStartSLOduration=4.795422553 podStartE2EDuration="4.795422553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.791877987 +0000 UTC m=+2186.381077049" watchObservedRunningTime="2026-11-01 23:00:04.795422553 +0000 UTC m=+2186.384621658"
Nov 01 23:00:05.107944 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.107911    4837 generic.go:356] "Generic (PLEG): container finished" podID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerID="f12c817eda7e7dafb08cc692246012554965b37b4c0c64e4b49e415b631c4a7c" exitCode=60
Nov 01 23:00:05.108398 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.107991    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" event={"ID":"5a841a30-8ea9-449e-bda1-1309b98b5141","Type":"ContainerDied","Data":"f12c817eda7e7dafb08cc692246012554965b37b4c0c64e4b49e415b631c4a7c"}
Nov 01 23:00:05.108398 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.108089    4837 scope.go:122] "RemoveContainer" containerID="11500f6f21540074dac3c02891996fa386bc621b3e2a423f8c71a7313a5e4ff6"
Nov 01 23:00:05.108398 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.108294    4837 scope.go:122] "RemoveContainer" containerID="f12c817eda7e7dafb08cc692246012554965b37b4c0c64e4b49e415b631c4a7c"
Nov 01 23:00:05.108650 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.108632    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn_odh-ai-gateway-infra(5a841a30-8ea9-449e-bda1-1309b98b5141)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141"
Nov 01 23:00:05.110639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.110616    4837 generic.go:356] "Generic (PLEG): container finished" podID="fabd9a7a-987c-47b2-9138-49889b712d21" containerID="8e475cf166b9cd9d198aec67d4e38d315192c5e77f45f41fb14553176dc2eae1" exitCode=60
Nov 01 23:00:05.110755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.110703    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" event={"ID":"fabd9a7a-987c-47b2-9138-49889b712d21","Type":"ContainerDied","Data":"8e475cf166b9cd9d198aec67d4e38d315192c5e77f45f41fb14553176dc2eae1"}
Nov 01 23:00:05.110957 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.110942    4837 scope.go:122] "RemoveContainer" containerID="8e475cf166b9cd9d198aec67d4e38d315192c5e77f45f41fb14553176dc2eae1"
Nov 01 23:00:05.111276 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.111257    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb_odh-ai-gateway-infra(fabd9a7a-987c-47b2-9138-49889b712d21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" podUID="fabd9a7a-987c-47b2-9138-49889b712d21"
Nov 01 23:00:05.113056 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.113024    4837 generic.go:356] "Generic (PLEG): container finished" podID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerID="c56925acecdaa70b71e629baa93baab7018fbf1b17ea7e18ab42cfc9446cdaf8" exitCode=60
Nov 01 23:00:05.113145 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.113063    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" event={"ID":"d6af8618-cbd0-4678-9dfc-088e8337ed08","Type":"ContainerDied","Data":"c56925acecdaa70b71e629baa93baab7018fbf1b17ea7e18ab42cfc9446cdaf8"}
Nov 01 23:00:05.113372 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.113358    4837 scope.go:122] "RemoveContainer" containerID="c56925acecdaa70b71e629baa93baab7018fbf1b17ea7e18ab42cfc9446cdaf8"
Nov 01 23:00:05.113675 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.113656    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn_odh-ai-gateway-infra(d6af8618-cbd0-4678-9dfc-088e8337ed08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08"
Nov 01 23:00:05.115257 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.115237    4837 generic.go:356] "Generic (PLEG): container finished" podID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerID="c0d42afd6d4ba7ab049b5d1396a3f1076f92901eafc2a989d5be79f46392d617" exitCode=60
Nov 01 23:00:05.115344 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.115309    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" event={"ID":"0172cd03-fd27-4b92-920b-ba2ee1b89f90","Type":"ContainerDied","Data":"c0d42afd6d4ba7ab049b5d1396a3f1076f92901eafc2a989d5be79f46392d617"}
Nov 01 23:00:05.115587 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.115571    4837 scope.go:122] "RemoveContainer" containerID="c0d42afd6d4ba7ab049b5d1396a3f1076f92901eafc2a989d5be79f46392d617"
Nov 01 23:00:05.115874 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.115856    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd_odh-ai-gateway-infra(0172cd03-fd27-4b92-920b-ba2ee1b89f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90"
Nov 01 23:00:05.117272 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.117254    4837 generic.go:356] "Generic (PLEG): container finished" podID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" containerID="a654c38ced2382e744d28b47973a79c78ca7c3364920b0683d8f6287652e1c04" exitCode=60
Nov 01 23:00:05.117371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.117351    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf" event={"ID":"f8d859c5-be57-4861-b33f-ecc6cb4d4178","Type":"ContainerDied","Data":"a654c38ced2382e744d28b47973a79c78ca7c3364920b0683d8f6287652e1c04"}
Nov 01 23:00:05.117520 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.117506    4837 scope.go:122] "RemoveContainer" containerID="a654c38ced2382e744d28b47973a79c78ca7c3364920b0683d8f6287652e1c04"
Nov 01 23:00:05.117773 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.117756    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf_odh-ai-gateway-infra(f8d859c5-be57-4861-b33f-ecc6cb4d4178)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178"
Nov 01 23:00:05.119141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.119123    4837 generic.go:356] "Generic (PLEG): container finished" podID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerID="b18d11d748898966b3a6be298a951922daf0d76245acd96ec34c193158201ec9" exitCode=60
Nov 01 23:00:05.119220 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.119198    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" event={"ID":"75034634-c95f-4cb2-aa56-897dc81cef3f","Type":"ContainerDied","Data":"b18d11d748898966b3a6be298a951922daf0d76245acd96ec34c193158201ec9"}
Nov 01 23:00:05.119393 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.119378    4837 scope.go:122] "RemoveContainer" containerID="b18d11d748898966b3a6be298a951922daf0d76245acd96ec34c193158201ec9"
Nov 01 23:00:05.119651 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.119632    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-sfj9q_odh-ai-gateway-infra(75034634-c95f-4cb2-aa56-897dc81cef3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f"
Nov 01 23:00:05.121162 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.121140    4837 generic.go:356] "Generic (PLEG): container finished" podID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerID="cba1f730d707e8366643ffe2ce0e508cfd89c7b47eb8c8ee3fa3e8ea4da80265" exitCode=60
Nov 01 23:00:05.121255 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.121223    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" event={"ID":"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5","Type":"ContainerDied","Data":"cba1f730d707e8366643ffe2ce0e508cfd89c7b47eb8c8ee3fa3e8ea4da80265"}
Nov 01 23:00:05.121741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.121419    4837 scope.go:122] "RemoveContainer" containerID="cba1f730d707e8366643ffe2ce0e508cfd89c7b47eb8c8ee3fa3e8ea4da80265"
Nov 01 23:00:05.121741 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.121694    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp_odh-ai-gateway-infra(4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5"
Nov 01 23:00:05.123389 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.123367    4837 generic.go:356] "Generic (PLEG): container finished" podID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerID="7bbf60c35ef66ffea54197be6e48769751c251f624c7f2afe52f6019550e6578" exitCode=60
Nov 01 23:00:05.123484 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.123446    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" event={"ID":"ea5787d4-6244-4b0f-a2ac-84044b041586","Type":"ContainerDied","Data":"7bbf60c35ef66ffea54197be6e48769751c251f624c7f2afe52f6019550e6578"}
Nov 01 23:00:05.123603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.123588    4837 scope.go:122] "RemoveContainer" containerID="7bbf60c35ef66ffea54197be6e48769751c251f624c7f2afe52f6019550e6578"
Nov 01 23:00:05.123872 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.123839    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh_odh-ai-gateway-infra(ea5787d4-6244-4b0f-a2ac-84044b041586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586"
Nov 01 23:00:05.123945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.123894    4837 scope.go:122] "RemoveContainer" containerID="3ee6c4b3c309fe235573f85525ed27bb04606f6c6810175195d1eab7f4ef8f5a"
Nov 01 23:00:05.125615 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.125596    4837 generic.go:356] "Generic (PLEG): container finished" podID="8c17bfba-c48b-41de-872a-c491be361d55" containerID="50268675d0e8253b92724ad796affac4cdbe8d1c3e30414fbdd57e85f9cf17ad" exitCode=60
Nov 01 23:00:05.125712 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.125672    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" event={"ID":"8c17bfba-c48b-41de-872a-c491be361d55","Type":"ContainerDied","Data":"50268675d0e8253b92724ad796affac4cdbe8d1c3e30414fbdd57e85f9cf17ad"}
Nov 01 23:00:05.125778 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.125715    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" podStartSLOduration=5.125700397 podStartE2EDuration="5.125700397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:05.120678891 +0000 UTC m=+2186.709877952" watchObservedRunningTime="2026-11-01 23:00:05.125700397 +0000 UTC m=+2186.714899459"
Nov 01 23:00:05.126001 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.125985    4837 scope.go:122] "RemoveContainer" containerID="50268675d0e8253b92724ad796affac4cdbe8d1c3e30414fbdd57e85f9cf17ad"
Nov 01 23:00:05.126254 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.126233    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9_odh-ai-gateway-infra(8c17bfba-c48b-41de-872a-c491be361d55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" podUID="8c17bfba-c48b-41de-872a-c491be361d55"
Nov 01 23:00:05.127473 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.127455    4837 generic.go:356] "Generic (PLEG): container finished" podID="3f78edcf-5daf-47ba-85f0-050cf11657fb" containerID="964134efec7dbfecec914489139577f2c4d612b2273dcacd40a605abb9e9db57" exitCode=60
Nov 01 23:00:05.127548 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.127513    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w" event={"ID":"3f78edcf-5daf-47ba-85f0-050cf11657fb","Type":"ContainerDied","Data":"964134efec7dbfecec914489139577f2c4d612b2273dcacd40a605abb9e9db57"}
Nov 01 23:00:05.127677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.127663    4837 scope.go:122] "RemoveContainer" containerID="964134efec7dbfecec914489139577f2c4d612b2273dcacd40a605abb9e9db57"
Nov 01 23:00:05.127903 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:05.127885    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w_odh-ai-gateway-infra(3f78edcf-5daf-47ba-85f0-050cf11657fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb"
Nov 01 23:00:05.137795 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.137775    4837 scope.go:122] "RemoveContainer" containerID="e86c13d6233c804c8b8df8b15ac3b7a635671b8f1017a6799f400b49fa5c7f58"
Nov 01 23:00:05.150922 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.150905    4837 scope.go:122] "RemoveContainer" containerID="d3387a252e033eb20aadf30de9a97d9793449de42817e666fb81f14cbaf989f3"
Nov 01 23:00:05.163967 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.163951    4837 scope.go:122] "RemoveContainer" containerID="56803d2d0e85f0415e4fa1e871a8c9fb19a26534910630c47ff18fea72f37e89"
Nov 01 23:00:05.178073 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.178030    4837 scope.go:122] "RemoveContainer" containerID="4c5f8594ad874bb2f0b3a56336efec8fa12b5d05b9b216c6d65d3ac04025812f"
Nov 01 23:00:05.192070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.192033    4837 scope.go:122] "RemoveContainer" containerID="9037f37c02b06f2731a7d9f2073b65fa761d21bd5f08d5199ad10e5cbbf4fc8a"
Nov 01 23:00:05.205475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.205452    4837 scope.go:122] "RemoveContainer" containerID="5c26e816a32a3917fa345921c44c99ea7c3d2541017685d774e78a63b1d66921"
Nov 01 23:00:05.218302 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.218283    4837 scope.go:122] "RemoveContainer" containerID="f31e33ed40b2983cc1c670a214d079707b4665a0e38853ec4535c8abfd55009b"
Nov 01 23:00:05.230791 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.230775    4837 scope.go:122] "RemoveContainer" containerID="54c71d19faba770f4656c0b40d74016866fc6370951b4c1f731cf4ab5ca2e725"
Nov 01 23:00:05.236864 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:05.236825    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf" podStartSLOduration=5.236813929 podStartE2EDuration="5.236813929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:05.23585508 +0000 UTC m=+2186.825054144" watchObservedRunningTime="2026-11-01 23:00:05.236813929 +0000 UTC m=+2186.826012989"
Nov 01 23:00:06.075082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.075035    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"]
Nov 01 23:00:06.141456 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.141426    4837 scope.go:122] "RemoveContainer" containerID="b18d11d748898966b3a6be298a951922daf0d76245acd96ec34c193158201ec9"
Nov 01 23:00:06.141896 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.141795    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-sfj9q_odh-ai-gateway-infra(75034634-c95f-4cb2-aa56-897dc81cef3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f"
Nov 01 23:00:06.143306 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.143285    4837 scope.go:122] "RemoveContainer" containerID="cba1f730d707e8366643ffe2ce0e508cfd89c7b47eb8c8ee3fa3e8ea4da80265"
Nov 01 23:00:06.144024 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.143671    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp_odh-ai-gateway-infra(4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5"
Nov 01 23:00:06.144916 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.144899    4837 scope.go:122] "RemoveContainer" containerID="7bbf60c35ef66ffea54197be6e48769751c251f624c7f2afe52f6019550e6578"
Nov 01 23:00:06.145310 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.145286    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh_odh-ai-gateway-infra(ea5787d4-6244-4b0f-a2ac-84044b041586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586"
Nov 01 23:00:06.147098 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.146618    4837 scope.go:122] "RemoveContainer" containerID="50268675d0e8253b92724ad796affac4cdbe8d1c3e30414fbdd57e85f9cf17ad"
Nov 01 23:00:06.147098 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.147004    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9_odh-ai-gateway-infra(8c17bfba-c48b-41de-872a-c491be361d55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" podUID="8c17bfba-c48b-41de-872a-c491be361d55"
Nov 01 23:00:06.160185 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.149301    4837 scope.go:122] "RemoveContainer" containerID="964134efec7dbfecec914489139577f2c4d612b2273dcacd40a605abb9e9db57"
Nov 01 23:00:06.160185 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.149728    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w_odh-ai-gateway-infra(3f78edcf-5daf-47ba-85f0-050cf11657fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb"
Nov 01 23:00:06.160185 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.156236    4837 scope.go:122] "RemoveContainer" containerID="f12c817eda7e7dafb08cc692246012554965b37b4c0c64e4b49e415b631c4a7c"
Nov 01 23:00:06.160185 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.157314    4837 scope.go:122] "RemoveContainer" containerID="8e475cf166b9cd9d198aec67d4e38d315192c5e77f45f41fb14553176dc2eae1"
Nov 01 23:00:06.160185 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.157739    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn_odh-ai-gateway-infra(5a841a30-8ea9-449e-bda1-1309b98b5141)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141"
Nov 01 23:00:06.160185 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.157741    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb_odh-ai-gateway-infra(fabd9a7a-987c-47b2-9138-49889b712d21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" podUID="fabd9a7a-987c-47b2-9138-49889b712d21"
Nov 01 23:00:06.171359 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.170611    4837 scope.go:122] "RemoveContainer" containerID="c56925acecdaa70b71e629baa93baab7018fbf1b17ea7e18ab42cfc9446cdaf8"
Nov 01 23:00:06.171359 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.171090    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn_odh-ai-gateway-infra(d6af8618-cbd0-4678-9dfc-088e8337ed08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08"
Nov 01 23:00:06.172769 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.172743    4837 scope.go:122] "RemoveContainer" containerID="c0d42afd6d4ba7ab049b5d1396a3f1076f92901eafc2a989d5be79f46392d617"
Nov 01 23:00:06.173215 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.173149    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd_odh-ai-gateway-infra(0172cd03-fd27-4b92-920b-ba2ee1b89f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90"
Nov 01 23:00:06.178915 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:06.178896    4837 scope.go:122] "RemoveContainer" containerID="a654c38ced2382e744d28b47973a79c78ca7c3364920b0683d8f6287652e1c04"
Nov 01 23:00:06.179299 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:06.179271    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf_odh-ai-gateway-infra(f8d859c5-be57-4861-b33f-ecc6cb4d4178)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178"
Nov 01 23:00:07.375167 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:07.375142    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:07.489543 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:07.489465    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d859c5-be57-4861-b33f-ecc6cb4d4178-kube-api-access-vtbs9\" (UniqueName: \"kubernetes.io/projected/f8d859c5-be57-4861-b33f-ecc6cb4d4178-kube-api-access-vtbs9\") pod \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\" (UID: \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\") "
Nov 01 23:00:07.489683 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:07.489645    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d859c5-be57-4861-b33f-ecc6cb4d4178-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d859c5-be57-4861-b33f-ecc6cb4d4178-maas-api-service-ca\") pod \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\" (UID: \"f8d859c5-be57-4861-b33f-ecc6cb4d4178\") "
Nov 01 23:00:07.489977 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:07.489955    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d859c5-be57-4861-b33f-ecc6cb4d4178-maas-api-service-ca" pod "f8d859c5-be57-4861-b33f-ecc6cb4d4178" (UID: "f8d859c5-be57-4861-b33f-ecc6cb4d4178"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:07.490309 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:07.490289    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d859c5-be57-4861-b33f-ecc6cb4d4178-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:07.491790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:07.491763    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d859c5-be57-4861-b33f-ecc6cb4d4178-kube-api-access-vtbs9" pod "f8d859c5-be57-4861-b33f-ecc6cb4d4178" (UID: "f8d859c5-be57-4861-b33f-ecc6cb4d4178"). InnerVolumeSpecName "kube-api-access-vtbs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:07.592325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:07.592292    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtbs9\" (UniqueName: \"kubernetes.io/projected/f8d859c5-be57-4861-b33f-ecc6cb4d4178-kube-api-access-vtbs9\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:08.195283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.195254    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"
Nov 01 23:00:08.195433 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.195294    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf" event={"ID":"f8d859c5-be57-4861-b33f-ecc6cb4d4178","Type":"ContainerDied","Data":"3a59c120cec808dbc243d71987df84239e61c2027293e94fd766f32a486f6c7b"}
Nov 01 23:00:08.195433 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.195359    4837 scope.go:122] "RemoveContainer" containerID="a654c38ced2382e744d28b47973a79c78ca7c3364920b0683d8f6287652e1c04"
Nov 01 23:00:08.362175 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.361132    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"]
Nov 01 23:00:08.397111 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.395926    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f0206-29892900-72qbf"]
Nov 01 23:00:08.526961 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.526894    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" path="/var/lib/kubelet/pods/f8d859c5-be57-4861-b33f-ecc6cb4d4178/volumes"
Nov 01 23:00:08.610128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.610094    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"]
Nov 01 23:00:08.610357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.610333    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy" containerID="cri-o://fd9766dddd371d31dd4fe13be28a5cced98c90874be3cc439f86eab3cbb0870f" gracePeriod=30
Nov 01 23:00:08.634626 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.634600    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"]
Nov 01 23:00:08.634816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:08.634796    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy" containerID="cri-o://d8cad0fb4fb3691853420f1d32c041b31fd1d983ea69ccb3c7737a60481d7540" gracePeriod=30
Nov 01 23:00:13.598560 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:13.598530    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"]
Nov 01 23:00:13.598968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:13.598750    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl" podUID="ec8f4047-d714-4652-8d8d-7cb2bf0202ce" containerName="maas-api" containerID="cri-o://b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3" gracePeriod=30
Nov 01 23:00:13.827218 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:13.827183    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"]
Nov 01 23:00:14.148246 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.143179    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 23:00:14.150838 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.150763    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-kube-api-access-mx997\" (UniqueName: \"kubernetes.io/projected/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-kube-api-access-mx997\") pod \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") "
Nov 01 23:00:14.151323 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.151296    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-metrics-certs\") pod \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") "
Nov 01 23:00:14.151528 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.151512    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-maas-api-tls\") pod \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") "
Nov 01 23:00:14.151709 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.151694    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-openshift-service-ca\") pod \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\" (UID: \"ec8f4047-d714-4652-8d8d-7cb2bf0202ce\") "
Nov 01 23:00:14.152920 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.152888    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-openshift-service-ca" pod "ec8f4047-d714-4652-8d8d-7cb2bf0202ce" (UID: "ec8f4047-d714-4652-8d8d-7cb2bf0202ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.158415 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.158139    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-metrics-certs" pod "ec8f4047-d714-4652-8d8d-7cb2bf0202ce" (UID: "ec8f4047-d714-4652-8d8d-7cb2bf0202ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:14.197074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.196371    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-maas-api-tls" pod "ec8f4047-d714-4652-8d8d-7cb2bf0202ce" (UID: "ec8f4047-d714-4652-8d8d-7cb2bf0202ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:14.267505 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.259652    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-kube-api-access-mx997" pod "ec8f4047-d714-4652-8d8d-7cb2bf0202ce" (UID: "ec8f4047-d714-4652-8d8d-7cb2bf0202ce"). InnerVolumeSpecName "kube-api-access-mx997". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.267505 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.259849    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.267505 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.259872    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx997\" (UniqueName: \"kubernetes.io/projected/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-kube-api-access-mx997\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.267505 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.259888    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.267505 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.259903    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8f4047-d714-4652-8d8d-7cb2bf0202ce-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.286375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.286332    4837 generic.go:356] "Generic (PLEG): container finished" podID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerID="fd9766dddd371d31dd4fe13be28a5cced98c90874be3cc439f86eab3cbb0870f" exitCode=0
Nov 01 23:00:14.286540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.286454    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" event={"ID":"979e8cc3-6842-4ea6-98cd-4c6c4ac52495","Type":"ContainerDied","Data":"fd9766dddd371d31dd4fe13be28a5cced98c90874be3cc439f86eab3cbb0870f"}
Nov 01 23:00:14.288760 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.288728    4837 generic.go:356] "Generic (PLEG): container finished" podID="ec8f4047-d714-4652-8d8d-7cb2bf0202ce" containerID="b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3" exitCode=0
Nov 01 23:00:14.288994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.288930    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl" event={"ID":"ec8f4047-d714-4652-8d8d-7cb2bf0202ce","Type":"ContainerDied","Data":"b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3"}
Nov 01 23:00:14.289079 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.289007    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl" event={"ID":"ec8f4047-d714-4652-8d8d-7cb2bf0202ce","Type":"ContainerDied","Data":"7a44a6be53b2ffc6388434d7c939ebe422349b4d01d2f9b0ed77204b8958050e"}
Nov 01 23:00:14.289168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.289119    4837 scope.go:122] "RemoveContainer" containerID="b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3"
Nov 01 23:00:14.289383 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.289367    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"
Nov 01 23:00:14.294390 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.294371    4837 generic.go:356] "Generic (PLEG): container finished" podID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerID="d8cad0fb4fb3691853420f1d32c041b31fd1d983ea69ccb3c7737a60481d7540" exitCode=0
Nov 01 23:00:14.294521 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.294509    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" event={"ID":"2f09d829-20e3-4dee-9a0d-005e944488a9","Type":"ContainerDied","Data":"d8cad0fb4fb3691853420f1d32c041b31fd1d983ea69ccb3c7737a60481d7540"}
Nov 01 23:00:14.303536 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.303510    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 23:00:14.305592 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.305574    4837 scope.go:122] "RemoveContainer" containerID="b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3"
Nov 01 23:00:14.305870 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:14.305841    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3\": container with ID starting with b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3 not found: ID does not exist" containerID="b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3"
Nov 01 23:00:14.305957 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.305880    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3"} err="failed to get container status \"b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3\": rpc error: code = NotFound desc = could not find container \"b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3\": container with ID starting with b72d27b5047e19e0e4021d4fdc7c7f498bb9fb0a05fdd738cf2273abfa76bda3 not found: ID does not exist"
Nov 01 23:00:14.305957 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.305904    4837 scope.go:122] "RemoveContainer" containerID="5c0869dcac0aed0c435b2788436e9b602fadfc8c393a37fc7f8a0cfa1240a781"
Nov 01 23:00:14.311458 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.311438    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 23:00:14.361109 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361082    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-token\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-token\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.361309 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361115    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-envoy\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.361309 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361151    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-podinfo\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.361309 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361201    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-socket\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.361309 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361231    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-podinfo\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.361546 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361335    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-certs\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.361546 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361398    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istiod-ca-cert\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.361546 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361444    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-kube-api-access-8qf4t\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-kube-api-access-8qf4t\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.361546 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361477    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-socket\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.361546 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361509    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-kube-api-access-gdsnz\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-kube-api-access-gdsnz\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.361797 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361550    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-credential-socket\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.361797 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361608    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-data\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.361797 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361708    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-envoy\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.361797 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361722    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-certs" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.361797 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361753    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-certs\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.362117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361812    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-token\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-token\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.362117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.361904    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-credential-socket\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.362117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.362023    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-socket" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.362117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.362060    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f09d829-20e3-4dee-9a0d-005e944488a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f09d829-20e3-4dee-9a0d-005e944488a9-istiod-ca-cert\") pod \"2f09d829-20e3-4dee-9a0d-005e944488a9\" (UID: \"2f09d829-20e3-4dee-9a0d-005e944488a9\") "
Nov 01 23:00:14.362331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.362137    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-data\") pod \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\" (UID: \"979e8cc3-6842-4ea6-98cd-4c6c4ac52495\") "
Nov 01 23:00:14.362331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.362270    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-data" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.362501 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.362479    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-socket" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.362728 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.362700    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-data" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.363140 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.363118    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.363227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.363147    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.363227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.363153    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-credential-socket" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.363227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.363163    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.363227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.363178    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.363227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.363193    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.363497 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.363404    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-credential-socket" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.363733 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.363710    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f09d829-20e3-4dee-9a0d-005e944488a9-istiod-ca-cert" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.364109 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.364085    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istiod-ca-cert" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.366562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.366538    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-envoy" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.366801 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.366779    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-certs" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.366915 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.366887    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-envoy" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.369412 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.369386    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-kube-api-access-gdsnz" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "kube-api-access-gdsnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.370419 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.370396    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-podinfo" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:14.370525 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.370427    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-kube-api-access-8qf4t" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "kube-api-access-8qf4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.370525 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.370467    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-token" pod "2f09d829-20e3-4dee-9a0d-005e944488a9" (UID: "2f09d829-20e3-4dee-9a0d-005e944488a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.371093 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.371037    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-token" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.371826 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.371799    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-podinfo" pod "979e8cc3-6842-4ea6-98cd-4c6c4ac52495" (UID: "979e8cc3-6842-4ea6-98cd-4c6c4ac52495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:14.393374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.393356    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:14.477617 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.476290    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"]
Nov 01 23:00:14.478066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478005    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478089    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478108    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478122    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478138    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f09d829-20e3-4dee-9a0d-005e944488a9-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478152    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478164    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478176    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f09d829-20e3-4dee-9a0d-005e944488a9-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478189    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478202    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478215    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qf4t\" (UniqueName: \"kubernetes.io/projected/979e8cc3-6842-4ea6-98cd-4c6c4ac52495-kube-api-access-8qf4t\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478230    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdsnz\" (UniqueName: \"kubernetes.io/projected/2f09d829-20e3-4dee-9a0d-005e944488a9-kube-api-access-gdsnz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.478609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.478244    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f09d829-20e3-4dee-9a0d-005e944488a9-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.491098 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.491059    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d68bab-5964d4746b-ck9wl"]
Nov 01 23:00:14.518337 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.518307    4837 scope.go:122] "RemoveContainer" containerID="50268675d0e8253b92724ad796affac4cdbe8d1c3e30414fbdd57e85f9cf17ad"
Nov 01 23:00:14.532502 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.532473    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8f4047-d714-4652-8d8d-7cb2bf0202ce" path="/var/lib/kubelet/pods/ec8f4047-d714-4652-8d8d-7cb2bf0202ce/volumes"
Nov 01 23:00:14.579563 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.579537    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f78edcf-5daf-47ba-85f0-050cf11657fb-kube-api-access-nzxlt\" (UniqueName: \"kubernetes.io/projected/3f78edcf-5daf-47ba-85f0-050cf11657fb-kube-api-access-nzxlt\") pod \"3f78edcf-5daf-47ba-85f0-050cf11657fb\" (UID: \"3f78edcf-5daf-47ba-85f0-050cf11657fb\") "
Nov 01 23:00:14.580340 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.580311    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f78edcf-5daf-47ba-85f0-050cf11657fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f78edcf-5daf-47ba-85f0-050cf11657fb-maas-api-service-ca\") pod \"3f78edcf-5daf-47ba-85f0-050cf11657fb\" (UID: \"3f78edcf-5daf-47ba-85f0-050cf11657fb\") "
Nov 01 23:00:14.580706 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.580682    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f78edcf-5daf-47ba-85f0-050cf11657fb-maas-api-service-ca" pod "3f78edcf-5daf-47ba-85f0-050cf11657fb" (UID: "3f78edcf-5daf-47ba-85f0-050cf11657fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.581249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.581216    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f78edcf-5daf-47ba-85f0-050cf11657fb-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.594422 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.594393    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f78edcf-5daf-47ba-85f0-050cf11657fb-kube-api-access-nzxlt" pod "3f78edcf-5daf-47ba-85f0-050cf11657fb" (UID: "3f78edcf-5daf-47ba-85f0-050cf11657fb"). InnerVolumeSpecName "kube-api-access-nzxlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.682702 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:14.682673    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzxlt\" (UniqueName: \"kubernetes.io/projected/3f78edcf-5daf-47ba-85f0-050cf11657fb-kube-api-access-nzxlt\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:15.308942 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.308911    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"
Nov 01 23:00:15.309141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.308955    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w" event={"ID":"2f09d829-20e3-4dee-9a0d-005e944488a9","Type":"ContainerDied","Data":"43aa79585de921d3a4b318c271bbd7de63eac07474b6e7ea654363545ad807ca"}
Nov 01 23:00:15.309141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.309026    4837 scope.go:122] "RemoveContainer" containerID="d8cad0fb4fb3691853420f1d32c041b31fd1d983ea69ccb3c7737a60481d7540"
Nov 01 23:00:15.312966 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.312683    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" event={"ID":"8c17bfba-c48b-41de-872a-c491be361d55","Type":"ContainerStarted","Data":"0c1e3dbfc2f90d618b3e658962d62929c0a1e4749e7145cb07d66bc394028cfc"}
Nov 01 23:00:15.328114 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.328090    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"
Nov 01 23:00:15.328232 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.328090    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w" event={"ID":"3f78edcf-5daf-47ba-85f0-050cf11657fb","Type":"ContainerDied","Data":"c6028276d5d5c42a0787eb609978855f91de84e8f5f22c5c67dc1ec3ea6cf4a3"}
Nov 01 23:00:15.329968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.329940    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn" event={"ID":"979e8cc3-6842-4ea6-98cd-4c6c4ac52495","Type":"ContainerDied","Data":"8ab5cfee830f60731e0078ff60c4853da8f036dce3c96470801869693fa40278"}
Nov 01 23:00:15.330110 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.330029    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"
Nov 01 23:00:15.339442 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.339422    4837 scope.go:122] "RemoveContainer" containerID="964134efec7dbfecec914489139577f2c4d612b2273dcacd40a605abb9e9db57"
Nov 01 23:00:15.351217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.351195    4837 scope.go:122] "RemoveContainer" containerID="fd9766dddd371d31dd4fe13be28a5cced98c90874be3cc439f86eab3cbb0870f"
Nov 01 23:00:15.517883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.517855    4837 scope.go:122] "RemoveContainer" containerID="b18d11d748898966b3a6be298a951922daf0d76245acd96ec34c193158201ec9"
Nov 01 23:00:15.643136 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.643100    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"]
Nov 01 23:00:15.649011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.648979    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-hcg2w"]
Nov 01 23:00:15.797083 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.786209    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"]
Nov 01 23:00:15.797083 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.787778    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-9dctn"]
Nov 01 23:00:15.867146 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.867110    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"]
Nov 01 23:00:15.885268 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:15.885241    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d68bab-29892900-h8t5w"]
Nov 01 23:00:16.138795 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.138716    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"]
Nov 01 23:00:16.138969 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.138937    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy" containerID="cri-o://0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2" gracePeriod=30
Nov 01 23:00:16.449071 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.446531    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" event={"ID":"75034634-c95f-4cb2-aa56-897dc81cef3f","Type":"ContainerStarted","Data":"3abbc7a3644029fe8216cfeded57f116c99281b7a1892dc1267b048edb0bb92e"}
Nov 01 23:00:16.468122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.462112    4837 generic.go:356] "Generic (PLEG): container finished" podID="8c17bfba-c48b-41de-872a-c491be361d55" containerID="0c1e3dbfc2f90d618b3e658962d62929c0a1e4749e7145cb07d66bc394028cfc" exitCode=60
Nov 01 23:00:16.468122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.462302    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" event={"ID":"8c17bfba-c48b-41de-872a-c491be361d55","Type":"ContainerDied","Data":"0c1e3dbfc2f90d618b3e658962d62929c0a1e4749e7145cb07d66bc394028cfc"}
Nov 01 23:00:16.468122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.462366    4837 scope.go:122] "RemoveContainer" containerID="50268675d0e8253b92724ad796affac4cdbe8d1c3e30414fbdd57e85f9cf17ad"
Nov 01 23:00:16.468122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.462912    4837 scope.go:122] "RemoveContainer" containerID="0c1e3dbfc2f90d618b3e658962d62929c0a1e4749e7145cb07d66bc394028cfc"
Nov 01 23:00:16.468122 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:16.463392    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9_odh-ai-gateway-infra(8c17bfba-c48b-41de-872a-c491be361d55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" podUID="8c17bfba-c48b-41de-872a-c491be361d55"
Nov 01 23:00:16.518359 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.518329    4837 scope.go:122] "RemoveContainer" containerID="cba1f730d707e8366643ffe2ce0e508cfd89c7b47eb8c8ee3fa3e8ea4da80265"
Nov 01 23:00:16.518508 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.518407    4837 scope.go:122] "RemoveContainer" containerID="8e475cf166b9cd9d198aec67d4e38d315192c5e77f45f41fb14553176dc2eae1"
Nov 01 23:00:16.518508 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.518476    4837 scope.go:122] "RemoveContainer" containerID="c56925acecdaa70b71e629baa93baab7018fbf1b17ea7e18ab42cfc9446cdaf8"
Nov 01 23:00:16.532671 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.532648    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" path="/var/lib/kubelet/pods/2f09d829-20e3-4dee-9a0d-005e944488a9/volumes"
Nov 01 23:00:16.533133 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.533117    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb" path="/var/lib/kubelet/pods/3f78edcf-5daf-47ba-85f0-050cf11657fb/volumes"
Nov 01 23:00:16.533582 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.533569    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" path="/var/lib/kubelet/pods/979e8cc3-6842-4ea6-98cd-4c6c4ac52495/volumes"
Nov 01 23:00:16.545771 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:16.545749    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"]
Nov 01 23:00:17.488854 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.488817    4837 generic.go:356] "Generic (PLEG): container finished" podID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerID="3abbc7a3644029fe8216cfeded57f116c99281b7a1892dc1267b048edb0bb92e" exitCode=60
Nov 01 23:00:17.489289 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.488892    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" event={"ID":"75034634-c95f-4cb2-aa56-897dc81cef3f","Type":"ContainerDied","Data":"3abbc7a3644029fe8216cfeded57f116c99281b7a1892dc1267b048edb0bb92e"}
Nov 01 23:00:17.489289 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.488957    4837 scope.go:122] "RemoveContainer" containerID="b18d11d748898966b3a6be298a951922daf0d76245acd96ec34c193158201ec9"
Nov 01 23:00:17.489430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.489408    4837 scope.go:122] "RemoveContainer" containerID="3abbc7a3644029fe8216cfeded57f116c99281b7a1892dc1267b048edb0bb92e"
Nov 01 23:00:17.489808 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:17.489786    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-sfj9q_odh-ai-gateway-infra(75034634-c95f-4cb2-aa56-897dc81cef3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f"
Nov 01 23:00:17.491106 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.491085    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" event={"ID":"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5","Type":"ContainerStarted","Data":"076989826eacd36599122894dfc3bf313f21aacc3af48cfaff9b4c25866ca37b"}
Nov 01 23:00:17.494917 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.494895    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" event={"ID":"fabd9a7a-987c-47b2-9138-49889b712d21","Type":"ContainerStarted","Data":"d5b89c2ac01e18ada33ce59b2dc77d026ba2c0c6fe29a8e1f0aab889aca53d69"}
Nov 01 23:00:17.497151 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.497129    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" event={"ID":"d6af8618-cbd0-4678-9dfc-088e8337ed08","Type":"ContainerStarted","Data":"fb80f2bd0c1c6606902c7b12896b49a30d1424b73f979ee3688f9a6a83eee495"}
Nov 01 23:00:17.518143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.518123    4837 scope.go:122] "RemoveContainer" containerID="f12c817eda7e7dafb08cc692246012554965b37b4c0c64e4b49e415b631c4a7c"
Nov 01 23:00:17.548896 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.548872    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"]
Nov 01 23:00:17.549099 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.549060    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy" containerID="cri-o://aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd" gracePeriod=30
Nov 01 23:00:17.583748 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.583717    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"]
Nov 01 23:00:17.832914 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.832847    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:17.837602 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.837581    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c17bfba-c48b-41de-872a-c491be361d55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c17bfba-c48b-41de-872a-c491be361d55-maas-api-service-ca\") pod \"8c17bfba-c48b-41de-872a-c491be361d55\" (UID: \"8c17bfba-c48b-41de-872a-c491be361d55\") "
Nov 01 23:00:17.837718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.837705    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c17bfba-c48b-41de-872a-c491be361d55-kube-api-access-nz794\" (UniqueName: \"kubernetes.io/projected/8c17bfba-c48b-41de-872a-c491be361d55-kube-api-access-nz794\") pod \"8c17bfba-c48b-41de-872a-c491be361d55\" (UID: \"8c17bfba-c48b-41de-872a-c491be361d55\") "
Nov 01 23:00:17.837913 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.837888    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c17bfba-c48b-41de-872a-c491be361d55-maas-api-service-ca" pod "8c17bfba-c48b-41de-872a-c491be361d55" (UID: "8c17bfba-c48b-41de-872a-c491be361d55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:17.838165 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.838149    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c17bfba-c48b-41de-872a-c491be361d55-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:17.839769 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.839744    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c17bfba-c48b-41de-872a-c491be361d55-kube-api-access-nz794" pod "8c17bfba-c48b-41de-872a-c491be361d55" (UID: "8c17bfba-c48b-41de-872a-c491be361d55"). InnerVolumeSpecName "kube-api-access-nz794". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:17.939513 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:17.939486    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz794\" (UniqueName: \"kubernetes.io/projected/8c17bfba-c48b-41de-872a-c491be361d55-kube-api-access-nz794\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:18.373494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.373439    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"]
Nov 01 23:00:18.375199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375169    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy"
Nov 01 23:00:18.375199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375193    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375218    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375226    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375265    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375277    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375294    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375302    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375320    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375326    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375346    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375353    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375375    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:18.375396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375381    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:18.375909 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375496    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" containerName="cleanup"
Nov 01 23:00:18.375909 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375503    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" containerName="cleanup"
Nov 01 23:00:18.375909 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375526    4837 cpu_manager.go:414] "containerMap: removing container" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy"
Nov 01 23:00:18.375909 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375532    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy"
Nov 01 23:00:18.375909 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375552    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ec8f4047-d714-4652-8d8d-7cb2bf0202ce" containerName="maas-api"
Nov 01 23:00:18.375909 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375561    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8f4047-d714-4652-8d8d-7cb2bf0202ce" containerName="maas-api"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375912    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" containerName="cleanup"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375928    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="979e8cc3-6842-4ea6-98cd-4c6c4ac52495" containerName="istio-proxy"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375943    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8f4047-d714-4652-8d8d-7cb2bf0202ce" containerName="maas-api"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375965    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.375983    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.376002    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d859c5-be57-4861-b33f-ecc6cb4d4178" containerName="cleanup"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.376011    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f09d829-20e3-4dee-9a0d-005e944488a9" containerName="istio-proxy"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.376021    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb" containerName="cleanup"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.376038    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f78edcf-5daf-47ba-85f0-050cf11657fb" containerName="cleanup"
Nov 01 23:00:18.376201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.376081    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:18.381509 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.381471    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.432359 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.432333    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:00:18.435785 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.435760    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"]
Nov 01 23:00:18.445951 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.445920    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbl9b\" (UniqueName: \"kubernetes.io/projected/3269d234-2352-4493-819d-007ffca37155-kube-api-access-tbl9b\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.446115 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.445973    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3269d234-2352-4493-819d-007ffca37155-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.446115 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.445993    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3269d234-2352-4493-819d-007ffca37155-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.511210 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.511179    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" event={"ID":"5a841a30-8ea9-449e-bda1-1309b98b5141","Type":"ContainerStarted","Data":"df8ee9d00e41d0edb6a461378041bb12001da685beaa3ba3bd9175a5db6b2e59"}
Nov 01 23:00:18.513209 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.513183    4837 generic.go:356] "Generic (PLEG): container finished" podID="fabd9a7a-987c-47b2-9138-49889b712d21" containerID="d5b89c2ac01e18ada33ce59b2dc77d026ba2c0c6fe29a8e1f0aab889aca53d69" exitCode=60
Nov 01 23:00:18.513328 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.513253    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" event={"ID":"fabd9a7a-987c-47b2-9138-49889b712d21","Type":"ContainerDied","Data":"d5b89c2ac01e18ada33ce59b2dc77d026ba2c0c6fe29a8e1f0aab889aca53d69"}
Nov 01 23:00:18.513328 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.513318    4837 scope.go:122] "RemoveContainer" containerID="8e475cf166b9cd9d198aec67d4e38d315192c5e77f45f41fb14553176dc2eae1"
Nov 01 23:00:18.513701 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.513680    4837 scope.go:122] "RemoveContainer" containerID="d5b89c2ac01e18ada33ce59b2dc77d026ba2c0c6fe29a8e1f0aab889aca53d69"
Nov 01 23:00:18.514087 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:18.514061    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb_odh-ai-gateway-infra(fabd9a7a-987c-47b2-9138-49889b712d21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" podUID="fabd9a7a-987c-47b2-9138-49889b712d21"
Nov 01 23:00:18.515345 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.515327    4837 generic.go:356] "Generic (PLEG): container finished" podID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerID="fb80f2bd0c1c6606902c7b12896b49a30d1424b73f979ee3688f9a6a83eee495" exitCode=60
Nov 01 23:00:18.515427 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.515376    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" event={"ID":"d6af8618-cbd0-4678-9dfc-088e8337ed08","Type":"ContainerDied","Data":"fb80f2bd0c1c6606902c7b12896b49a30d1424b73f979ee3688f9a6a83eee495"}
Nov 01 23:00:18.515731 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.515716    4837 scope.go:122] "RemoveContainer" containerID="fb80f2bd0c1c6606902c7b12896b49a30d1424b73f979ee3688f9a6a83eee495"
Nov 01 23:00:18.516058 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:18.516023    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn_odh-ai-gateway-infra(d6af8618-cbd0-4678-9dfc-088e8337ed08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08"
Nov 01 23:00:18.517300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.517282    4837 scope.go:122] "RemoveContainer" containerID="c0d42afd6d4ba7ab049b5d1396a3f1076f92901eafc2a989d5be79f46392d617"
Nov 01 23:00:18.517563 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.517460    4837 scope.go:122] "RemoveContainer" containerID="7bbf60c35ef66ffea54197be6e48769751c251f624c7f2afe52f6019550e6578"
Nov 01 23:00:18.520941 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.520907    4837 generic.go:356] "Generic (PLEG): container finished" podID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerID="076989826eacd36599122894dfc3bf313f21aacc3af48cfaff9b4c25866ca37b" exitCode=60
Nov 01 23:00:18.525458 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.525427    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:18.536339 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.535502    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" event={"ID":"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5","Type":"ContainerDied","Data":"076989826eacd36599122894dfc3bf313f21aacc3af48cfaff9b4c25866ca37b"}
Nov 01 23:00:18.536339 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.535552    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" event={"ID":"8c17bfba-c48b-41de-872a-c491be361d55","Type":"ContainerDied","Data":"7042e9049b132e9e1f48413085adb8fa67a5e0fac3df3a85a0fbb27bd37006b1"}
Nov 01 23:00:18.536339 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.535918    4837 scope.go:122] "RemoveContainer" containerID="076989826eacd36599122894dfc3bf313f21aacc3af48cfaff9b4c25866ca37b"
Nov 01 23:00:18.536339 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:18.536191    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp_odh-ai-gateway-infra(4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5"
Nov 01 23:00:18.537923 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.537897    4837 scope.go:122] "RemoveContainer" containerID="c56925acecdaa70b71e629baa93baab7018fbf1b17ea7e18ab42cfc9446cdaf8"
Nov 01 23:00:18.547898 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.547868    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbl9b\" (UniqueName: \"kubernetes.io/projected/3269d234-2352-4493-819d-007ffca37155-kube-api-access-tbl9b\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.548011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.547949    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3269d234-2352-4493-819d-007ffca37155-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.548011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.547975    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3269d234-2352-4493-819d-007ffca37155-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.548950 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.548918    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3269d234-2352-4493-819d-007ffca37155-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.550768 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.550748    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3269d234-2352-4493-819d-007ffca37155-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.585415 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.585393    4837 scope.go:122] "RemoveContainer" containerID="cba1f730d707e8366643ffe2ce0e508cfd89c7b47eb8c8ee3fa3e8ea4da80265"
Nov 01 23:00:18.604531 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.604464    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbl9b\" (UniqueName: \"kubernetes.io/projected/3269d234-2352-4493-819d-007ffca37155-kube-api-access-tbl9b\") pod \"maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.613533 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.612803    4837 scope.go:122] "RemoveContainer" containerID="0c1e3dbfc2f90d618b3e658962d62929c0a1e4749e7145cb07d66bc394028cfc"
Nov 01 23:00:18.698213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.698182    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:18.795208 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.795175    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:18.836501 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:18.836450    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:00:18.874799 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.874576    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75034634-c95f-4cb2-aa56-897dc81cef3f-kube-api-access-pt48s\" (UniqueName: \"kubernetes.io/projected/75034634-c95f-4cb2-aa56-897dc81cef3f-kube-api-access-pt48s\") pod \"75034634-c95f-4cb2-aa56-897dc81cef3f\" (UID: \"75034634-c95f-4cb2-aa56-897dc81cef3f\") "
Nov 01 23:00:18.874799 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.874673    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75034634-c95f-4cb2-aa56-897dc81cef3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75034634-c95f-4cb2-aa56-897dc81cef3f-maas-api-service-ca\") pod \"75034634-c95f-4cb2-aa56-897dc81cef3f\" (UID: \"75034634-c95f-4cb2-aa56-897dc81cef3f\") "
Nov 01 23:00:18.886381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.886300    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75034634-c95f-4cb2-aa56-897dc81cef3f-maas-api-service-ca" pod "75034634-c95f-4cb2-aa56-897dc81cef3f" (UID: "75034634-c95f-4cb2-aa56-897dc81cef3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:18.886381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.886364    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"]
Nov 01 23:00:18.891477 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.891444    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75034634-c95f-4cb2-aa56-897dc81cef3f-kube-api-access-pt48s" pod "75034634-c95f-4cb2-aa56-897dc81cef3f" (UID: "75034634-c95f-4cb2-aa56-897dc81cef3f"). InnerVolumeSpecName "kube-api-access-pt48s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:18.968483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.968458    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"]
Nov 01 23:00:18.976730 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.976706    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt48s\" (UniqueName: \"kubernetes.io/projected/75034634-c95f-4cb2-aa56-897dc81cef3f-kube-api-access-pt48s\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:18.976730 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:18.976733    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75034634-c95f-4cb2-aa56-897dc81cef3f-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:19.089010 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.088977    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"]
Nov 01 23:00:19.107699 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.107677    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"]
Nov 01 23:00:19.107847 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:19.107760    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3269d234_2352_4493_819d_007ffca37155.slice/crio-034b566476148be9225072a1eba96f62a2f4645830b9065d7e7490a93518ee50 WatchSource:0}: Error finding container 034b566476148be9225072a1eba96f62a2f4645830b9065d7e7490a93518ee50: Status 404 returned error can't find the container with id 034b566476148be9225072a1eba96f62a2f4645830b9065d7e7490a93518ee50
Nov 01 23:00:19.536513 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.536486    4837 generic.go:356] "Generic (PLEG): container finished" podID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerID="df8ee9d00e41d0edb6a461378041bb12001da685beaa3ba3bd9175a5db6b2e59" exitCode=60
Nov 01 23:00:19.536868 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.536565    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" event={"ID":"5a841a30-8ea9-449e-bda1-1309b98b5141","Type":"ContainerDied","Data":"df8ee9d00e41d0edb6a461378041bb12001da685beaa3ba3bd9175a5db6b2e59"}
Nov 01 23:00:19.536868 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.536645    4837 scope.go:122] "RemoveContainer" containerID="f12c817eda7e7dafb08cc692246012554965b37b4c0c64e4b49e415b631c4a7c"
Nov 01 23:00:19.537121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.537103    4837 scope.go:122] "RemoveContainer" containerID="df8ee9d00e41d0edb6a461378041bb12001da685beaa3ba3bd9175a5db6b2e59"
Nov 01 23:00:19.537626 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:19.537491    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn_odh-ai-gateway-infra(5a841a30-8ea9-449e-bda1-1309b98b5141)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141"
Nov 01 23:00:19.543001 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.542979    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" event={"ID":"0172cd03-fd27-4b92-920b-ba2ee1b89f90","Type":"ContainerStarted","Data":"e4e7009fdff06f27ac6d5cdec2f87845825dc52f102aa19990e66101c554d2d6"}
Nov 01 23:00:19.543476 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.543457    4837 scope.go:122] "RemoveContainer" containerID="e4e7009fdff06f27ac6d5cdec2f87845825dc52f102aa19990e66101c554d2d6"
Nov 01 23:00:19.543874 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:19.543838    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd_odh-ai-gateway-infra(0172cd03-fd27-4b92-920b-ba2ee1b89f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90"
Nov 01 23:00:19.544746 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.544690    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z" event={"ID":"3269d234-2352-4493-819d-007ffca37155","Type":"ContainerStarted","Data":"034b566476148be9225072a1eba96f62a2f4645830b9065d7e7490a93518ee50"}
Nov 01 23:00:19.546240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.546214    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"
Nov 01 23:00:19.546240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.546228    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q" event={"ID":"75034634-c95f-4cb2-aa56-897dc81cef3f","Type":"ContainerDied","Data":"0e8af44db2628afceefac4605ae5fe6863a757f2ccdb4271f45888052e369442"}
Nov 01 23:00:19.551919 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.551892    4837 generic.go:356] "Generic (PLEG): container finished" podID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerID="1cea21ae8d475882988938e625eb1d02eae9edec8c788b016b33b14c23f0eefb" exitCode=60
Nov 01 23:00:19.552025 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.551993    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" event={"ID":"ea5787d4-6244-4b0f-a2ac-84044b041586","Type":"ContainerDied","Data":"1cea21ae8d475882988938e625eb1d02eae9edec8c788b016b33b14c23f0eefb"}
Nov 01 23:00:19.552409 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.552388    4837 scope.go:122] "RemoveContainer" containerID="1cea21ae8d475882988938e625eb1d02eae9edec8c788b016b33b14c23f0eefb"
Nov 01 23:00:19.552733 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:19.552708    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh_odh-ai-gateway-infra(ea5787d4-6244-4b0f-a2ac-84044b041586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586"
Nov 01 23:00:19.598208 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.598181    4837 scope.go:122] "RemoveContainer" containerID="3abbc7a3644029fe8216cfeded57f116c99281b7a1892dc1267b048edb0bb92e"
Nov 01 23:00:19.610535 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.610508    4837 scope.go:122] "RemoveContainer" containerID="7bbf60c35ef66ffea54197be6e48769751c251f624c7f2afe52f6019550e6578"
Nov 01 23:00:19.670436 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.670405    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"]
Nov 01 23:00:19.760064 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.760019    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:19.817211 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.817186    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:19.819483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.819460    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:19.893530 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.893491    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabd9a7a-987c-47b2-9138-49889b712d21-kube-api-access-4r9zb\" (UniqueName: \"kubernetes.io/projected/fabd9a7a-987c-47b2-9138-49889b712d21-kube-api-access-4r9zb\") pod \"fabd9a7a-987c-47b2-9138-49889b712d21\" (UID: \"fabd9a7a-987c-47b2-9138-49889b712d21\") "
Nov 01 23:00:19.893530 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.893532    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-kube-api-access-2nbg4\" (UniqueName: \"kubernetes.io/projected/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-kube-api-access-2nbg4\") pod \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\" (UID: \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\") "
Nov 01 23:00:19.893799 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.893652    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-maas-api-service-ca\") pod \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\" (UID: \"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5\") "
Nov 01 23:00:19.893799 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.893789    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6af8618-cbd0-4678-9dfc-088e8337ed08-kube-api-access-9665q\" (UniqueName: \"kubernetes.io/projected/d6af8618-cbd0-4678-9dfc-088e8337ed08-kube-api-access-9665q\") pod \"d6af8618-cbd0-4678-9dfc-088e8337ed08\" (UID: \"d6af8618-cbd0-4678-9dfc-088e8337ed08\") "
Nov 01 23:00:19.893906 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.893832    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fabd9a7a-987c-47b2-9138-49889b712d21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd9a7a-987c-47b2-9138-49889b712d21-maas-api-service-ca\") pod \"fabd9a7a-987c-47b2-9138-49889b712d21\" (UID: \"fabd9a7a-987c-47b2-9138-49889b712d21\") "
Nov 01 23:00:19.893906 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.893857    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6af8618-cbd0-4678-9dfc-088e8337ed08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6af8618-cbd0-4678-9dfc-088e8337ed08-maas-api-service-ca\") pod \"d6af8618-cbd0-4678-9dfc-088e8337ed08\" (UID: \"d6af8618-cbd0-4678-9dfc-088e8337ed08\") "
Nov 01 23:00:19.894125 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.893998    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-maas-api-service-ca" pod "4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" (UID: "4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:19.894217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.894200    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabd9a7a-987c-47b2-9138-49889b712d21-maas-api-service-ca" pod "fabd9a7a-987c-47b2-9138-49889b712d21" (UID: "fabd9a7a-987c-47b2-9138-49889b712d21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:19.894312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.894287    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6af8618-cbd0-4678-9dfc-088e8337ed08-maas-api-service-ca" pod "d6af8618-cbd0-4678-9dfc-088e8337ed08" (UID: "d6af8618-cbd0-4678-9dfc-088e8337ed08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:19.894713 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.894690    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fabd9a7a-987c-47b2-9138-49889b712d21-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:19.894800 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.894719    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6af8618-cbd0-4678-9dfc-088e8337ed08-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:19.894800 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.894735    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:19.895858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.895830    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabd9a7a-987c-47b2-9138-49889b712d21-kube-api-access-4r9zb" pod "fabd9a7a-987c-47b2-9138-49889b712d21" (UID: "fabd9a7a-987c-47b2-9138-49889b712d21"). InnerVolumeSpecName "kube-api-access-4r9zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:19.895980 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.895962    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6af8618-cbd0-4678-9dfc-088e8337ed08-kube-api-access-9665q" pod "d6af8618-cbd0-4678-9dfc-088e8337ed08" (UID: "d6af8618-cbd0-4678-9dfc-088e8337ed08"). InnerVolumeSpecName "kube-api-access-9665q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:19.896028 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.896011    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-kube-api-access-2nbg4" pod "4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" (UID: "4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5"). InnerVolumeSpecName "kube-api-access-2nbg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:19.978141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.978064    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"]
Nov 01 23:00:20.007092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.994440    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-sfj9q"]
Nov 01 23:00:20.007092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.996730    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9665q\" (UniqueName: \"kubernetes.io/projected/d6af8618-cbd0-4678-9dfc-088e8337ed08-kube-api-access-9665q\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.007092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.996756    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r9zb\" (UniqueName: \"kubernetes.io/projected/fabd9a7a-987c-47b2-9138-49889b712d21-kube-api-access-4r9zb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.007092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:19.996771    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nbg4\" (UniqueName: \"kubernetes.io/projected/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5-kube-api-access-2nbg4\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.531248 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.531218    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" path="/var/lib/kubelet/pods/75034634-c95f-4cb2-aa56-897dc81cef3f/volumes"
Nov 01 23:00:20.564871 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.564838    4837 generic.go:356] "Generic (PLEG): container finished" podID="3269d234-2352-4493-819d-007ffca37155" containerID="4f34ddccdfabd8a81c95906e13e951ef21e89c6c34721aa8e30ac09f2476e75a" exitCode=0
Nov 01 23:00:20.565321 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.564981    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z" event={"ID":"3269d234-2352-4493-819d-007ffca37155","Type":"ContainerDied","Data":"4f34ddccdfabd8a81c95906e13e951ef21e89c6c34721aa8e30ac09f2476e75a"}
Nov 01 23:00:20.568210 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.568178    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp" event={"ID":"4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5","Type":"ContainerDied","Data":"6f22cfb8e3512d9a919e008aa52bd74c9855fee1c0008332972aff26353b7fad"}
Nov 01 23:00:20.568331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.568261    4837 scope.go:122] "RemoveContainer" containerID="076989826eacd36599122894dfc3bf313f21aacc3af48cfaff9b4c25866ca37b"
Nov 01 23:00:20.568826 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.568604    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"
Nov 01 23:00:20.577495 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.577471    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"
Nov 01 23:00:20.577604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.577467    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb" event={"ID":"fabd9a7a-987c-47b2-9138-49889b712d21","Type":"ContainerDied","Data":"df800499b4c7150041bd26309d15d05aee22df27ad744d9b222faa6605c879af"}
Nov 01 23:00:20.595119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.594155    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"
Nov 01 23:00:20.595119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.594591    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn" event={"ID":"d6af8618-cbd0-4678-9dfc-088e8337ed08","Type":"ContainerDied","Data":"dc083da79831d63b3945eb1f093155faa8d543e7a8921fd6fab627b1483135b4"}
Nov 01 23:00:20.597769 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.597743    4837 generic.go:356] "Generic (PLEG): container finished" podID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerID="e4e7009fdff06f27ac6d5cdec2f87845825dc52f102aa19990e66101c554d2d6" exitCode=60
Nov 01 23:00:20.597876 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.597781    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" event={"ID":"0172cd03-fd27-4b92-920b-ba2ee1b89f90","Type":"ContainerDied","Data":"e4e7009fdff06f27ac6d5cdec2f87845825dc52f102aa19990e66101c554d2d6"}
Nov 01 23:00:20.598507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.598488    4837 scope.go:122] "RemoveContainer" containerID="e4e7009fdff06f27ac6d5cdec2f87845825dc52f102aa19990e66101c554d2d6"
Nov 01 23:00:20.598889 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:20.598865    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd_odh-ai-gateway-infra(0172cd03-fd27-4b92-920b-ba2ee1b89f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90"
Nov 01 23:00:20.607270 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.607253    4837 scope.go:122] "RemoveContainer" containerID="d5b89c2ac01e18ada33ce59b2dc77d026ba2c0c6fe29a8e1f0aab889aca53d69"
Nov 01 23:00:20.616752 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.616732    4837 scope.go:122] "RemoveContainer" containerID="fb80f2bd0c1c6606902c7b12896b49a30d1424b73f979ee3688f9a6a83eee495"
Nov 01 23:00:20.626951 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.626929    4837 scope.go:122] "RemoveContainer" containerID="c0d42afd6d4ba7ab049b5d1396a3f1076f92901eafc2a989d5be79f46392d617"
Nov 01 23:00:20.805085 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.805010    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:20.884952 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.884918    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"]
Nov 01 23:00:20.910975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.910948    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a841a30-8ea9-449e-bda1-1309b98b5141-kube-api-access-58mhg\" (UniqueName: \"kubernetes.io/projected/5a841a30-8ea9-449e-bda1-1309b98b5141-kube-api-access-58mhg\") pod \"5a841a30-8ea9-449e-bda1-1309b98b5141\" (UID: \"5a841a30-8ea9-449e-bda1-1309b98b5141\") "
Nov 01 23:00:20.911155 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.911013    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a841a30-8ea9-449e-bda1-1309b98b5141-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a841a30-8ea9-449e-bda1-1309b98b5141-maas-api-service-ca\") pod \"5a841a30-8ea9-449e-bda1-1309b98b5141\" (UID: \"5a841a30-8ea9-449e-bda1-1309b98b5141\") "
Nov 01 23:00:20.911431 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.911391    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a841a30-8ea9-449e-bda1-1309b98b5141-maas-api-service-ca" pod "5a841a30-8ea9-449e-bda1-1309b98b5141" (UID: "5a841a30-8ea9-449e-bda1-1309b98b5141"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:20.911669 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.911647    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a841a30-8ea9-449e-bda1-1309b98b5141-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.913750 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:20.913723    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a841a30-8ea9-449e-bda1-1309b98b5141-kube-api-access-58mhg" pod "5a841a30-8ea9-449e-bda1-1309b98b5141" (UID: "5a841a30-8ea9-449e-bda1-1309b98b5141"). InnerVolumeSpecName "kube-api-access-58mhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:21.013598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.013565    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mhg\" (UniqueName: \"kubernetes.io/projected/5a841a30-8ea9-449e-bda1-1309b98b5141-kube-api-access-58mhg\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.064666 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.064581    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"]
Nov 01 23:00:21.070124 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.070096    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892900-qdvzb"]
Nov 01 23:00:21.298073 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.298002    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"]
Nov 01 23:00:21.315324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.315082    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"]
Nov 01 23:00:21.346101 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.344609    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-646881-29892900-n4jzn"]
Nov 01 23:00:21.425156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.425065    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 23:00:21.438409 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.438382    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"]
Nov 01 23:00:21.444614 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.444570    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9dfe9f-29892900-xbmkp"]
Nov 01 23:00:21.499182 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.499157    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:21.537677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.537653    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-envoy\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.537822 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.537726    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-data\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.537822 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.537772    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-token\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-token\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.537822 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.537818    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-credential-socket\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.537962 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.537837    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-socket\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.537962 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.537854    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-kube-api-access-swk49\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-kube-api-access-swk49\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.537962 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.537905    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istiod-ca-cert\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.537962 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.537937    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-certs\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.538182 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538000    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-data" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:21.538182 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538067    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-podinfo\") pod \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\" (UID: \"8603b6a4-30a6-4b97-b2b9-2d0f80c74365\") "
Nov 01 23:00:21.538182 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538112    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-credential-socket" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:21.538319 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538297    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-certs" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:21.538374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538362    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-socket" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:21.538571 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538540    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istiod-ca-cert" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:21.538908 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538888    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.538983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538916    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.538983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538931    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.538983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538944    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.538983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.538972    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.540243 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.540222    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-envoy" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:21.540371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.540351    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-kube-api-access-swk49" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "kube-api-access-swk49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:21.540538 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.540517    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-podinfo" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:21.540576 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.540550    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-token" pod "8603b6a4-30a6-4b97-b2b9-2d0f80c74365" (UID: "8603b6a4-30a6-4b97-b2b9-2d0f80c74365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:21.607239 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.607209    4837 generic.go:356] "Generic (PLEG): container finished" podID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerID="0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2" exitCode=0
Nov 01 23:00:21.607641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.607275    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"
Nov 01 23:00:21.607641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.607289    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" event={"ID":"8603b6a4-30a6-4b97-b2b9-2d0f80c74365","Type":"ContainerDied","Data":"0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2"}
Nov 01 23:00:21.607641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.607351    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57" event={"ID":"8603b6a4-30a6-4b97-b2b9-2d0f80c74365","Type":"ContainerDied","Data":"989091eb8d703b1a7e08b9cdda2332b326d2b6595577192099badff3c5b840f3"}
Nov 01 23:00:21.607641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.607390    4837 scope.go:122] "RemoveContainer" containerID="0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2"
Nov 01 23:00:21.616973 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.616941    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"
Nov 01 23:00:21.617118 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.616997    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh" event={"ID":"ea5787d4-6244-4b0f-a2ac-84044b041586","Type":"ContainerDied","Data":"3a24352a7c82259db4511fcfd73590b62afc5a46d3c2e7afb69af768ae7a3e5a"}
Nov 01 23:00:21.619694 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.619616    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"
Nov 01 23:00:21.619694 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.619660    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn" event={"ID":"5a841a30-8ea9-449e-bda1-1309b98b5141","Type":"ContainerDied","Data":"428ed9d663eb8ff885ec1f966732f3109be9f6649ab501bf9eaaece0fee6ee33"}
Nov 01 23:00:21.624194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.624174    4837 scope.go:122] "RemoveContainer" containerID="9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2"
Nov 01 23:00:21.635813 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.635792    4837 scope.go:122] "RemoveContainer" containerID="0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2"
Nov 01 23:00:21.636115 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:21.636087    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2\": container with ID starting with 0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2 not found: ID does not exist" containerID="0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2"
Nov 01 23:00:21.636227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.636128    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2"} err="failed to get container status \"0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2\": rpc error: code = NotFound desc = could not find container \"0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2\": container with ID starting with 0b24864aa52a1e11cbf2e97dcaf24bf09ac2be3c2e4cd046a70f0a3dc3c8c5a2 not found: ID does not exist"
Nov 01 23:00:21.636227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.636156    4837 scope.go:122] "RemoveContainer" containerID="9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2"
Nov 01 23:00:21.636498 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:21.636460    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2\": container with ID starting with 9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2 not found: ID does not exist" containerID="9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2"
Nov 01 23:00:21.636587 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.636493    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2"} err="failed to get container status \"9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2\": rpc error: code = NotFound desc = could not find container \"9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2\": container with ID starting with 9d7d415acc8f3a847ada57cf6f9c2a28d58c093a7314f9e1e52c674cc34887f2 not found: ID does not exist"
Nov 01 23:00:21.637156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.636514    4837 scope.go:122] "RemoveContainer" containerID="1cea21ae8d475882988938e625eb1d02eae9edec8c788b016b33b14c23f0eefb"
Nov 01 23:00:21.640413 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.640379    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5787d4-6244-4b0f-a2ac-84044b041586-kube-api-access-dn9m5\" (UniqueName: \"kubernetes.io/projected/ea5787d4-6244-4b0f-a2ac-84044b041586-kube-api-access-dn9m5\") pod \"ea5787d4-6244-4b0f-a2ac-84044b041586\" (UID: \"ea5787d4-6244-4b0f-a2ac-84044b041586\") "
Nov 01 23:00:21.640515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.640472    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5787d4-6244-4b0f-a2ac-84044b041586-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5787d4-6244-4b0f-a2ac-84044b041586-maas-api-service-ca\") pod \"ea5787d4-6244-4b0f-a2ac-84044b041586\" (UID: \"ea5787d4-6244-4b0f-a2ac-84044b041586\") "
Nov 01 23:00:21.641438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.641419    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.641438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.641440    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swk49\" (UniqueName: \"kubernetes.io/projected/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-kube-api-access-swk49\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.641601 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.641449    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.641601 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.641459    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603b6a4-30a6-4b97-b2b9-2d0f80c74365-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.642678 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.642637    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5787d4-6244-4b0f-a2ac-84044b041586-maas-api-service-ca" pod "ea5787d4-6244-4b0f-a2ac-84044b041586" (UID: "ea5787d4-6244-4b0f-a2ac-84044b041586"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:21.643823 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.643798    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5787d4-6244-4b0f-a2ac-84044b041586-kube-api-access-dn9m5" pod "ea5787d4-6244-4b0f-a2ac-84044b041586" (UID: "ea5787d4-6244-4b0f-a2ac-84044b041586"). InnerVolumeSpecName "kube-api-access-dn9m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:21.649310 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.649289    4837 scope.go:122] "RemoveContainer" containerID="df8ee9d00e41d0edb6a461378041bb12001da685beaa3ba3bd9175a5db6b2e59"
Nov 01 23:00:21.743677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.743592    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5787d4-6244-4b0f-a2ac-84044b041586-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.743677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.743625    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn9m5\" (UniqueName: \"kubernetes.io/projected/ea5787d4-6244-4b0f-a2ac-84044b041586-kube-api-access-dn9m5\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.850130 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.850106    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:21.852600 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.852582    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:21.888986 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.888953    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"]
Nov 01 23:00:21.896620 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.896597    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-e71d55-29892900-642dn"]
Nov 01 23:00:21.945576 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.945553    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3269d234-2352-4493-819d-007ffca37155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3269d234-2352-4493-819d-007ffca37155-maas-api-xks-ca\") pod \"3269d234-2352-4493-819d-007ffca37155\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") "
Nov 01 23:00:21.945697 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.945615    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3269d234-2352-4493-819d-007ffca37155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3269d234-2352-4493-819d-007ffca37155-maas-api-openshift-ca\") pod \"3269d234-2352-4493-819d-007ffca37155\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") "
Nov 01 23:00:21.945697 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.945646    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0172cd03-fd27-4b92-920b-ba2ee1b89f90-kube-api-access-gn8dl\" (UniqueName: \"kubernetes.io/projected/0172cd03-fd27-4b92-920b-ba2ee1b89f90-kube-api-access-gn8dl\") pod \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\" (UID: \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\") "
Nov 01 23:00:21.945807 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.945731    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0172cd03-fd27-4b92-920b-ba2ee1b89f90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0172cd03-fd27-4b92-920b-ba2ee1b89f90-maas-api-service-ca\") pod \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\" (UID: \"0172cd03-fd27-4b92-920b-ba2ee1b89f90\") "
Nov 01 23:00:21.945957 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.945932    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3269d234-2352-4493-819d-007ffca37155-kube-api-access-tbl9b\" (UniqueName: \"kubernetes.io/projected/3269d234-2352-4493-819d-007ffca37155-kube-api-access-tbl9b\") pod \"3269d234-2352-4493-819d-007ffca37155\" (UID: \"3269d234-2352-4493-819d-007ffca37155\") "
Nov 01 23:00:21.946125 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.946063    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3269d234-2352-4493-819d-007ffca37155-maas-api-openshift-ca" pod "3269d234-2352-4493-819d-007ffca37155" (UID: "3269d234-2352-4493-819d-007ffca37155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:21.946192 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.946158    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0172cd03-fd27-4b92-920b-ba2ee1b89f90-maas-api-service-ca" pod "0172cd03-fd27-4b92-920b-ba2ee1b89f90" (UID: "0172cd03-fd27-4b92-920b-ba2ee1b89f90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:21.946756 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.946731    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0172cd03-fd27-4b92-920b-ba2ee1b89f90-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.946853 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.946763    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3269d234-2352-4493-819d-007ffca37155-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.948123 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.948101    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3269d234-2352-4493-819d-007ffca37155-maas-api-xks-ca" pod "3269d234-2352-4493-819d-007ffca37155" (UID: "3269d234-2352-4493-819d-007ffca37155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:21.948263 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.948201    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0172cd03-fd27-4b92-920b-ba2ee1b89f90-kube-api-access-gn8dl" pod "0172cd03-fd27-4b92-920b-ba2ee1b89f90" (UID: "0172cd03-fd27-4b92-920b-ba2ee1b89f90"). InnerVolumeSpecName "kube-api-access-gn8dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:21.948330 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:21.948260    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3269d234-2352-4493-819d-007ffca37155-kube-api-access-tbl9b" pod "3269d234-2352-4493-819d-007ffca37155" (UID: "3269d234-2352-4493-819d-007ffca37155"). InnerVolumeSpecName "kube-api-access-tbl9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:22.048652 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.048618    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbl9b\" (UniqueName: \"kubernetes.io/projected/3269d234-2352-4493-819d-007ffca37155-kube-api-access-tbl9b\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:22.048652 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.048650    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3269d234-2352-4493-819d-007ffca37155-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:22.048855 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.048667    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn8dl\" (UniqueName: \"kubernetes.io/projected/0172cd03-fd27-4b92-920b-ba2ee1b89f90-kube-api-access-gn8dl\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:22.250128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.250093    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"]
Nov 01 23:00:22.253698 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.253668    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f0206-openshift-default-5488fc55f7-77p57"]
Nov 01 23:00:22.356418 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.356388    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"]
Nov 01 23:00:22.364721 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.364694    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892900-prlzh"]
Nov 01 23:00:22.628070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.627217    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" path="/var/lib/kubelet/pods/4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5/volumes"
Nov 01 23:00:22.628070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.627930    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" path="/var/lib/kubelet/pods/5a841a30-8ea9-449e-bda1-1309b98b5141/volumes"
Nov 01 23:00:22.633069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.632493    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" path="/var/lib/kubelet/pods/8603b6a4-30a6-4b97-b2b9-2d0f80c74365/volumes"
Nov 01 23:00:22.633315 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.633124    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" path="/var/lib/kubelet/pods/d6af8618-cbd0-4678-9dfc-088e8337ed08/volumes"
Nov 01 23:00:22.633743 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.633716    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" path="/var/lib/kubelet/pods/ea5787d4-6244-4b0f-a2ac-84044b041586/volumes"
Nov 01 23:00:22.646441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.645510    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" path="/var/lib/kubelet/pods/fabd9a7a-987c-47b2-9138-49889b712d21/volumes"
Nov 01 23:00:22.783326 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.783293    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd" event={"ID":"0172cd03-fd27-4b92-920b-ba2ee1b89f90","Type":"ContainerDied","Data":"15fc2801ac7521c232e6065204535ad0b65e7ed2b253330c99d43c61f454b8ba"}
Nov 01 23:00:22.783494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.783357    4837 scope.go:122] "RemoveContainer" containerID="e4e7009fdff06f27ac6d5cdec2f87845825dc52f102aa19990e66101c554d2d6"
Nov 01 23:00:22.783494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.783485    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"
Nov 01 23:00:22.789945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.789921    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z" event={"ID":"3269d234-2352-4493-819d-007ffca37155","Type":"ContainerDied","Data":"034b566476148be9225072a1eba96f62a2f4645830b9065d7e7490a93518ee50"}
Nov 01 23:00:22.790086 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.789943    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"
Nov 01 23:00:22.790086 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.789962    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="034b566476148be9225072a1eba96f62a2f4645830b9065d7e7490a93518ee50"
Nov 01 23:00:22.802937 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.802909    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"]
Nov 01 23:00:22.820870 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.820836    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"]
Nov 01 23:00:22.834937 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:22.834911    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"]
Nov 01 23:00:23.017557 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.017528    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 23:00:23.100773 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.100738    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"]
Nov 01 23:00:23.106512 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.106481    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8bac2f-29892900-4txnd"]
Nov 01 23:00:23.195504 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195476    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-socket\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.195671 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195597    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-credential-socket\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.195752 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195668    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-certs\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.195752 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195718    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f70dbb-d022-4392-b469-c9b518f8592d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f70dbb-d022-4392-b469-c9b518f8592d-istiod-ca-cert\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.195752 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195738    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38f70dbb-d022-4392-b469-c9b518f8592d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f70dbb-d022-4392-b469-c9b518f8592d-istio-podinfo\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.195953 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195763    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-kube-api-access-65cbl\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-kube-api-access-65cbl\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.195953 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195734    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-socket" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:23.196092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195944    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-credential-socket" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:23.196092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195926    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-certs" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:23.196092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.195962    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-data\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.196232 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196130    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-istio-token\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-istio-token\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.196350 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196239    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-envoy\") pod \"38f70dbb-d022-4392-b469-c9b518f8592d\" (UID: \"38f70dbb-d022-4392-b469-c9b518f8592d\") "
Nov 01 23:00:23.196350 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196249    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-data" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:23.196477 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196439    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f70dbb-d022-4392-b469-c9b518f8592d-istiod-ca-cert" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:23.196899 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196879    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.196899 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196899    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.197104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196910    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f70dbb-d022-4392-b469-c9b518f8592d-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.197104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196921    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.197104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.196929    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.198317 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.198287    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-istio-token" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:23.198431 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.198410    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-envoy" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:23.198586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.198547    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-kube-api-access-65cbl" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "kube-api-access-65cbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:23.198586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.198568    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38f70dbb-d022-4392-b469-c9b518f8592d-istio-podinfo" pod "38f70dbb-d022-4392-b469-c9b518f8592d" (UID: "38f70dbb-d022-4392-b469-c9b518f8592d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:23.298486 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.298393    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f70dbb-d022-4392-b469-c9b518f8592d-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.298486 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.298429    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65cbl\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-kube-api-access-65cbl\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.298486 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.298443    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f70dbb-d022-4392-b469-c9b518f8592d-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.298486 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.298458    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f70dbb-d022-4392-b469-c9b518f8592d-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:23.486251 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.486177    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"]
Nov 01 23:00:23.486394 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.486374    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy" containerID="cri-o://80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8" gracePeriod=30
Nov 01 23:00:23.852633 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.852513    4837 generic.go:356] "Generic (PLEG): container finished" podID="38f70dbb-d022-4392-b469-c9b518f8592d" containerID="aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd" exitCode=0
Nov 01 23:00:23.853312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.853283    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" event={"ID":"38f70dbb-d022-4392-b469-c9b518f8592d","Type":"ContainerDied","Data":"aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd"}
Nov 01 23:00:23.853463 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.853449    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf" event={"ID":"38f70dbb-d022-4392-b469-c9b518f8592d","Type":"ContainerDied","Data":"f00164ccf2eb5fd2980f10401c160ebdf5236d00f4a8f5ef52de467a07f00c64"}
Nov 01 23:00:23.853567 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.853556    4837 scope.go:122] "RemoveContainer" containerID="aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd"
Nov 01 23:00:23.853867 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.853813    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"
Nov 01 23:00:23.890573 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.890554    4837 scope.go:122] "RemoveContainer" containerID="aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd"
Nov 01 23:00:23.890858 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:23.890835    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd\": container with ID starting with aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd not found: ID does not exist" containerID="aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd"
Nov 01 23:00:23.890927 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:23.890873    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd"} err="failed to get container status \"aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd\": rpc error: code = NotFound desc = could not find container \"aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd\": container with ID starting with aa9d2002b9104994dcc3f6bfa5c106742d8a89bf59ec4b590212c792def8d6fd not found: ID does not exist"
Nov 01 23:00:24.065143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:24.060413    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"]
Nov 01 23:00:24.098316 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:24.098279    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-xr9lf"]
Nov 01 23:00:24.544153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:24.544110    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" path="/var/lib/kubelet/pods/0172cd03-fd27-4b92-920b-ba2ee1b89f90/volumes"
Nov 01 23:00:24.544725 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:24.544700    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" path="/var/lib/kubelet/pods/38f70dbb-d022-4392-b469-c9b518f8592d/volumes"
Nov 01 23:00:26.791266 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.791229    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"]
Nov 01 23:00:26.792181 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792164    4837 cpu_manager.go:414] "containerMap: removing container" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:26.792249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792182    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:26.792249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792205    4837 cpu_manager.go:414] "containerMap: removing container" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:26.792249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792213    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:26.792249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792224    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:26.792249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792231    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:26.792249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792245    4837 cpu_manager.go:414] "containerMap: removing container" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792252    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792271    4837 cpu_manager.go:414] "containerMap: removing container" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792278    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792294    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792301    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792309    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792317    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792337    4837 cpu_manager.go:414] "containerMap: removing container" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792344    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792358    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792364    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792376    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792382    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792391    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792398    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792412    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792419    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792431    4837 cpu_manager.go:414] "containerMap: removing container" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792437    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792448    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792454    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792465    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792470    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792478    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:26.792479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792485    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792499    4837 cpu_manager.go:414] "containerMap: removing container" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792505    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792515    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792521    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792533    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792539    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792552    4837 cpu_manager.go:414] "containerMap: removing container" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792558    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792569    4837 cpu_manager.go:414] "containerMap: removing container" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792575    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792585    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792591    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792600    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792606    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792624    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3269d234-2352-4493-819d-007ffca37155" containerName="revoke-keys"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792629    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3269d234-2352-4493-819d-007ffca37155" containerName="revoke-keys"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792786    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792798    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792809    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792818    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792828    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3269d234-2352-4493-819d-007ffca37155" containerName="revoke-keys"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792835    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792848    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792855    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792864    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c17bfba-c48b-41de-872a-c491be361d55" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792873    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792882    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792892    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792901    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792910    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792918    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f70dbb-d022-4392-b469-c9b518f8592d" containerName="istio-proxy"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792928    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8603b6a4-30a6-4b97-b2b9-2d0f80c74365" containerName="istio-proxy"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792937    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792945    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792954    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:26.793203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.792962    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d89a4be-f89e-4f7a-8c7f-5b4e18bb09c5" containerName="cleanup"
Nov 01 23:00:26.798491 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.798470    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.899989    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-socket\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.900072    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5463180-2162-4d32-b896-2ddd39e1b170-istiod-ca-cert\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.900115    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88dx\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-kube-api-access-x88dx\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.900194    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-data\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.900255    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-istio-token\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.900306    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-certs\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.901307    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-credential-socket\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.901348    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5463180-2162-4d32-b896-2ddd39e1b170-istio-podinfo\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.905066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.901382    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-envoy\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:26.973695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:26.930183    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"]
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025315    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-envoy\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025545    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-socket\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025583    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5463180-2162-4d32-b896-2ddd39e1b170-istiod-ca-cert\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025627    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88dx\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-kube-api-access-x88dx\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025712    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-data\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025771    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-istio-token\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025825    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-certs\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025855    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-credential-socket\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.026134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.025878    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5463180-2162-4d32-b896-2ddd39e1b170-istio-podinfo\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.041264 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.041235    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-675d1d-openshift-default-dockercfg-bxdbn\""
Nov 01 23:00:27.043397 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.043334    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-data\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.043888 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.043864    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-socket\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.044696 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.044675    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5463180-2162-4d32-b896-2ddd39e1b170-istiod-ca-cert\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.045358 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.045340    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-certs\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.046120 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.046100    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-credential-socket\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.052364 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.052340    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5463180-2162-4d32-b896-2ddd39e1b170-istio-podinfo\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.145783 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.142355    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-istio-token\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.145783 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.142866    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-envoy\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.181064 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.174982    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88dx\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-kube-api-access-x88dx\") pod \"e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") " pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.428366 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.428336    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:27.858298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.858273    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"]
Nov 01 23:00:27.858632 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:27.858364    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5463180_2162_4d32_b896_2ddd39e1b170.slice/crio-b9148b91d9d60abdb1fa35ee158c77e7f70c90279f696bb8f89579b6932bfa70 WatchSource:0}: Error finding container b9148b91d9d60abdb1fa35ee158c77e7f70c90279f696bb8f89579b6932bfa70: Status 404 returned error can't find the container with id b9148b91d9d60abdb1fa35ee158c77e7f70c90279f696bb8f89579b6932bfa70
Nov 01 23:00:27.861194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.861163    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:00:27.861284 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.861225    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:00:27.861284 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:27.861253    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:00:28.093536 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.093496    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" event={"ID":"b5463180-2162-4d32-b896-2ddd39e1b170","Type":"ContainerStarted","Data":"6c9787cb3e6af8a184d49c392282fe7d8c10a539e2b1d61790467708e0c75d96"}
Nov 01 23:00:28.093734 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.093561    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" event={"ID":"b5463180-2162-4d32-b896-2ddd39e1b170","Type":"ContainerStarted","Data":"b9148b91d9d60abdb1fa35ee158c77e7f70c90279f696bb8f89579b6932bfa70"}
Nov 01 23:00:28.207686 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.207624    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" podStartSLOduration=2.207605322 podStartE2EDuration="2.207605322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:28.204954973 +0000 UTC m=+2209.794154038" watchObservedRunningTime="2026-11-01 23:00:28.207605322 +0000 UTC m=+2209.796804384"
Nov 01 23:00:28.429454 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.429419    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:28.431543 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.431525    4837 patch_prober.go:29] interesting pod/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 01 23:00:28.431614 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.431596    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 01 23:00:28.768296 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.768234    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 23:00:28.858306 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858277    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-certs\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858317    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-istio-token\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-istio-token\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858337    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-data\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858459    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-kube-api-access-f9tx9\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-kube-api-access-f9tx9\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858494    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-socket\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858583    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e21ad7-e679-4381-a949-814c8b5207cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e21ad7-e679-4381-a949-814c8b5207cd-istiod-ca-cert\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858612    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-credential-socket\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858550    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-certs" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858684    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80e21ad7-e679-4381-a949-814c8b5207cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e21ad7-e679-4381-a949-814c8b5207cd-istio-podinfo\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858688    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-data" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858754    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-envoy\") pod \"80e21ad7-e679-4381-a949-814c8b5207cd\" (UID: \"80e21ad7-e679-4381-a949-814c8b5207cd\") "
Nov 01 23:00:28.858792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858790    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-socket" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:28.859312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858924    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e21ad7-e679-4381-a949-814c8b5207cd-istiod-ca-cert" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:28.859312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.858933    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-credential-socket" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:28.859402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.859388    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.859439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.859412    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.859439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.859427    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.859515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.859441    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e21ad7-e679-4381-a949-814c8b5207cd-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.859515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.859456    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.860657 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.860626    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-istio-token" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:28.860789 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.860753    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e21ad7-e679-4381-a949-814c8b5207cd-istio-podinfo" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:28.860930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.860913    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-kube-api-access-f9tx9" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "kube-api-access-f9tx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:28.861097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.861076    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-envoy" pod "80e21ad7-e679-4381-a949-814c8b5207cd" (UID: "80e21ad7-e679-4381-a949-814c8b5207cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:28.960532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.960488    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9tx9\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-kube-api-access-f9tx9\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.960532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.960526    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e21ad7-e679-4381-a949-814c8b5207cd-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.960532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.960536    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e21ad7-e679-4381-a949-814c8b5207cd-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.960532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:28.960545    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e21ad7-e679-4381-a949-814c8b5207cd-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.204061 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.200627    4837 generic.go:356] "Generic (PLEG): container finished" podID="80e21ad7-e679-4381-a949-814c8b5207cd" containerID="80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8" exitCode=0
Nov 01 23:00:29.204061 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.201642    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"
Nov 01 23:00:29.205794 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.205467    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" event={"ID":"80e21ad7-e679-4381-a949-814c8b5207cd","Type":"ContainerDied","Data":"80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8"}
Nov 01 23:00:29.205794 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.205548    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr" event={"ID":"80e21ad7-e679-4381-a949-814c8b5207cd","Type":"ContainerDied","Data":"3cec13261211e16c1f886b7ae4baf9939dc055581d8efd73a73bfa3daf5ed710"}
Nov 01 23:00:29.205794 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.205582    4837 scope.go:122] "RemoveContainer" containerID="80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8"
Nov 01 23:00:29.260714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.260379    4837 scope.go:122] "RemoveContainer" containerID="ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f"
Nov 01 23:00:29.276333 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.276306    4837 scope.go:122] "RemoveContainer" containerID="80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8"
Nov 01 23:00:29.276675 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:29.276647    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8\": container with ID starting with 80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8 not found: ID does not exist" containerID="80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8"
Nov 01 23:00:29.276787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.276683    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8"} err="failed to get container status \"80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8\": rpc error: code = NotFound desc = could not find container \"80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8\": container with ID starting with 80e574f8c479f214db6d92d7506119d34b36a588a49ddfe4705bd344bc0f07f8 not found: ID does not exist"
Nov 01 23:00:29.276787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.276705    4837 scope.go:122] "RemoveContainer" containerID="ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f"
Nov 01 23:00:29.277072 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:29.277020    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f\": container with ID starting with ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f not found: ID does not exist" containerID="ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f"
Nov 01 23:00:29.277122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.277083    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f"} err="failed to get container status \"ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f\": rpc error: code = NotFound desc = could not find container \"ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f\": container with ID starting with ae1da73696b991ecd2c311b1bdf9dbd0d2826c1affabc9d31ff2c0ccaf938d6f not found: ID does not exist"
Nov 01 23:00:29.378768 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.378738    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"]
Nov 01 23:00:29.382340 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.382317    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f5b935-openshift-default-5cd45bfb6c-6pxrr"]
Nov 01 23:00:29.433741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.433719    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:29.485121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:29.485019    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:00:30.211007 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:30.210976    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:30.211968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:30.211942    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:00:30.567899 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:30.567822    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" path="/var/lib/kubelet/pods/80e21ad7-e679-4381-a949-814c8b5207cd/volumes"
Nov 01 23:00:32.989065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:32.988819    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded"
Nov 01 23:00:33.057091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:33.015756    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:33.057659 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:33.043946    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:33.069375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:33.045426    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 23:00:33.594475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:33.575450    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:33.613035 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:33.612996    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:33.723217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:33.723168    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:34.007199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:34.007154    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:34.070066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:34.069321    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:34.135067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:34.115067    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"]
Nov 01 23:00:34.238068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:34.236762    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy" containerID="cri-o://eae58a75f219598fe97aa7a7b1367c424992baa8148da0bd08bd2da9e4d154cb" gracePeriod=30
Nov 01 23:00:35.755128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:35.755097    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"]
Nov 01 23:00:37.834650 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:37.834612    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" containerID="cri-o://9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b" gracePeriod=45
Nov 01 23:00:37.846456 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:37.846408    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main" containerID="cri-o://67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552" gracePeriod=45
Nov 01 23:00:37.860986 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:37.860958    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" containerID="cri-o://b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa" gracePeriod=45
Nov 01 23:00:37.904206 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:37.904179    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": dial tcp 10.134.0.78:8000: connect: connection refused"
Nov 01 23:00:37.989664 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:37.989492    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused"
Nov 01 23:00:38.188508 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.188142    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 23:00:38.190434 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.190405    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 23:00:38.198765 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.198741    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 23:00:38.235204 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235177    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-kube-api-access-hcwn4\" (UniqueName: \"kubernetes.io/projected/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-kube-api-access-hcwn4\") pod \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") "
Nov 01 23:00:38.235362 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235209    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-model-cache\") pod \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") "
Nov 01 23:00:38.235362 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235231    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-model-cache\") pod \"85be5c4e-2135-40a8-85eb-43a9021b30a2\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") "
Nov 01 23:00:38.235362 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235265    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-dshm\") pod \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") "
Nov 01 23:00:38.235362 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235359    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-tmp-dir\") pod \"e5890029-fa50-49f7-bf3f-0803dd176bcb\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") "
Nov 01 23:00:38.235569 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235394    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-home\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-home\") pod \"e5890029-fa50-49f7-bf3f-0803dd176bcb\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") "
Nov 01 23:00:38.235569 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235514    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-model-cache" pod "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" (UID: "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.235569 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235509    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-model-cache" pod "85be5c4e-2135-40a8-85eb-43a9021b30a2" (UID: "85be5c4e-2135-40a8-85eb-43a9021b30a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.235715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235662    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-home" pod "e5890029-fa50-49f7-bf3f-0803dd176bcb" (UID: "e5890029-fa50-49f7-bf3f-0803dd176bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.235715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235674    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-tmp-dir" pod "e5890029-fa50-49f7-bf3f-0803dd176bcb" (UID: "e5890029-fa50-49f7-bf3f-0803dd176bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.235715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235694    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85be5c4e-2135-40a8-85eb-43a9021b30a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/85be5c4e-2135-40a8-85eb-43a9021b30a2-tls-certs\") pod \"85be5c4e-2135-40a8-85eb-43a9021b30a2\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") "
Nov 01 23:00:38.235851 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235748    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tmp-dir\") pod \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") "
Nov 01 23:00:38.235851 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235788    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-dshm\") pod \"85be5c4e-2135-40a8-85eb-43a9021b30a2\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") "
Nov 01 23:00:38.235949 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.235848    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-dshm\") pod \"e5890029-fa50-49f7-bf3f-0803dd176bcb\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") "
Nov 01 23:00:38.236089 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.236036    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tmp-dir" pod "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" (UID: "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.236378 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.236314    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5890029-fa50-49f7-bf3f-0803dd176bcb-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890029-fa50-49f7-bf3f-0803dd176bcb-tls-certs\") pod \"e5890029-fa50-49f7-bf3f-0803dd176bcb\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") "
Nov 01 23:00:38.236492 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.236468    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-tmp-dir\") pod \"85be5c4e-2135-40a8-85eb-43a9021b30a2\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") "
Nov 01 23:00:38.237474 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.236809    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-tmp-dir" pod "85be5c4e-2135-40a8-85eb-43a9021b30a2" (UID: "85be5c4e-2135-40a8-85eb-43a9021b30a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.237655 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.237636    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85be5c4e-2135-40a8-85eb-43a9021b30a2-kube-api-access-786ns\" (UniqueName: \"kubernetes.io/projected/85be5c4e-2135-40a8-85eb-43a9021b30a2-kube-api-access-786ns\") pod \"85be5c4e-2135-40a8-85eb-43a9021b30a2\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") "
Nov 01 23:00:38.238189 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238172    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5890029-fa50-49f7-bf3f-0803dd176bcb-kube-api-access-j9s2m\" (UniqueName: \"kubernetes.io/projected/e5890029-fa50-49f7-bf3f-0803dd176bcb-kube-api-access-j9s2m\") pod \"e5890029-fa50-49f7-bf3f-0803dd176bcb\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") "
Nov 01 23:00:38.238489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238171    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-dshm" pod "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" (UID: "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.238489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238353    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-dshm" pod "85be5c4e-2135-40a8-85eb-43a9021b30a2" (UID: "85be5c4e-2135-40a8-85eb-43a9021b30a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.238489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238371    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-kube-api-access-hcwn4" pod "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" (UID: "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b"). InnerVolumeSpecName "kube-api-access-hcwn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:38.238788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238391    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-dshm" pod "e5890029-fa50-49f7-bf3f-0803dd176bcb" (UID: "e5890029-fa50-49f7-bf3f-0803dd176bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.238788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238443    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85be5c4e-2135-40a8-85eb-43a9021b30a2-tls-certs" pod "85be5c4e-2135-40a8-85eb-43a9021b30a2" (UID: "85be5c4e-2135-40a8-85eb-43a9021b30a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:38.238788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238486    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-home\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-home\") pod \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") "
Nov 01 23:00:38.238788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238587    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tls-certs\") pod \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\" (UID: \"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b\") "
Nov 01 23:00:38.238788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238689    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-model-cache\") pod \"e5890029-fa50-49f7-bf3f-0803dd176bcb\" (UID: \"e5890029-fa50-49f7-bf3f-0803dd176bcb\") "
Nov 01 23:00:38.238788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238714    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-home" pod "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" (UID: "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.238788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238724    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-home\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-home\") pod \"85be5c4e-2135-40a8-85eb-43a9021b30a2\" (UID: \"85be5c4e-2135-40a8-85eb-43a9021b30a2\") "
Nov 01 23:00:38.239240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.238928    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-model-cache" pod "e5890029-fa50-49f7-bf3f-0803dd176bcb" (UID: "e5890029-fa50-49f7-bf3f-0803dd176bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.239240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239019    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-home" pod "85be5c4e-2135-40a8-85eb-43a9021b30a2" (UID: "85be5c4e-2135-40a8-85eb-43a9021b30a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.239240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239187    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5890029-fa50-49f7-bf3f-0803dd176bcb-tls-certs" pod "e5890029-fa50-49f7-bf3f-0803dd176bcb" (UID: "e5890029-fa50-49f7-bf3f-0803dd176bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:38.239526 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239505    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239531    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239545    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcwn4\" (UniqueName: \"kubernetes.io/projected/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-kube-api-access-hcwn4\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239560    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239574    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239587    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239601    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239613    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239625    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85be5c4e-2135-40a8-85eb-43a9021b30a2-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239636    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239647    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239657    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5890029-fa50-49f7-bf3f-0803dd176bcb-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239668    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5890029-fa50-49f7-bf3f-0803dd176bcb-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239679    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85be5c4e-2135-40a8-85eb-43a9021b30a2-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.239945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.239689    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.240379 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.240307    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5890029-fa50-49f7-bf3f-0803dd176bcb-kube-api-access-j9s2m" pod "e5890029-fa50-49f7-bf3f-0803dd176bcb" (UID: "e5890029-fa50-49f7-bf3f-0803dd176bcb"). InnerVolumeSpecName "kube-api-access-j9s2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:38.240558 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.240537    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tls-certs" pod "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" (UID: "6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:38.240701 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.240685    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85be5c4e-2135-40a8-85eb-43a9021b30a2-kube-api-access-786ns" pod "85be5c4e-2135-40a8-85eb-43a9021b30a2" (UID: "85be5c4e-2135-40a8-85eb-43a9021b30a2"). InnerVolumeSpecName "kube-api-access-786ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:38.343137 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.341360    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-786ns\" (UniqueName: \"kubernetes.io/projected/85be5c4e-2135-40a8-85eb-43a9021b30a2-kube-api-access-786ns\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.343137 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.341394    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9s2m\" (UniqueName: \"kubernetes.io/projected/e5890029-fa50-49f7-bf3f-0803dd176bcb-kube-api-access-j9s2m\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.343137 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.341408    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.788428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.787072    4837 generic.go:356] "Generic (PLEG): container finished" podID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerID="9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b" exitCode=0
Nov 01 23:00:38.788428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.787183    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" event={"ID":"85be5c4e-2135-40a8-85eb-43a9021b30a2","Type":"ContainerDied","Data":"9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b"}
Nov 01 23:00:38.788428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.787223    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft" event={"ID":"85be5c4e-2135-40a8-85eb-43a9021b30a2","Type":"ContainerDied","Data":"c4bf48a3d487fe9809383d722c1c0bf99ace7b34a2ce3c0b72c356f492869625"}
Nov 01 23:00:38.788428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.787250    4837 scope.go:122] "RemoveContainer" containerID="9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b"
Nov 01 23:00:38.788428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.787406    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"
Nov 01 23:00:38.798060 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.798013    4837 generic.go:356] "Generic (PLEG): container finished" podID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerID="67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552" exitCode=0
Nov 01 23:00:38.801750 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.801685    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" event={"ID":"e5890029-fa50-49f7-bf3f-0803dd176bcb","Type":"ContainerDied","Data":"67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552"}
Nov 01 23:00:38.802011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.801975    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc" event={"ID":"e5890029-fa50-49f7-bf3f-0803dd176bcb","Type":"ContainerDied","Data":"6c4052ebc35cb5d5e3170197f135c2951d04e5cfab95c1a1a8c26aef2e46abf3"}
Nov 01 23:00:38.817101 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.806129    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"
Nov 01 23:00:38.825005 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.824970    4837 generic.go:356] "Generic (PLEG): container finished" podID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerID="b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa" exitCode=0
Nov 01 23:00:38.825453 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.825427    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" event={"ID":"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b","Type":"ContainerDied","Data":"b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa"}
Nov 01 23:00:38.825613 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.825598    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl" event={"ID":"6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b","Type":"ContainerDied","Data":"0fbc1ae183c4997441cc4b32259800a352ffffad3a05db4ff9d0c3aeacd5a33a"}
Nov 01 23:00:38.825793 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.825780    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"
Nov 01 23:00:38.836222 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.836201    4837 scope.go:122] "RemoveContainer" containerID="9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b"
Nov 01 23:00:38.837007 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:38.836974    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b\": container with ID starting with 9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b not found: ID does not exist" containerID="9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b"
Nov 01 23:00:38.837117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.837016    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b"} err="failed to get container status \"9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b\": rpc error: code = NotFound desc = could not find container \"9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b\": container with ID starting with 9c8db8188e760e40700777ba5c2a4406ec5305702f7b6f5a22c6450d34a3e20b not found: ID does not exist"
Nov 01 23:00:38.837117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.837059    4837 scope.go:122] "RemoveContainer" containerID="67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552"
Nov 01 23:00:38.848794 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.848771    4837 scope.go:122] "RemoveContainer" containerID="67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552"
Nov 01 23:00:38.849167 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:38.849145    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552\": container with ID starting with 67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552 not found: ID does not exist" containerID="67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552"
Nov 01 23:00:38.849229 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.849177    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552"} err="failed to get container status \"67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552\": rpc error: code = NotFound desc = could not find container \"67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552\": container with ID starting with 67901cd4518d56a78be1bca12e7bc7292a1eba40fb3f1ae7c19b6272b434c552 not found: ID does not exist"
Nov 01 23:00:38.849229 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.849195    4837 scope.go:122] "RemoveContainer" containerID="b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa"
Nov 01 23:00:38.875244 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.875223    4837 scope.go:122] "RemoveContainer" containerID="b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa"
Nov 01 23:00:38.875558 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:38.875536    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa\": container with ID starting with b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa not found: ID does not exist" containerID="b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa"
Nov 01 23:00:38.875611 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.875568    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa"} err="failed to get container status \"b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa\": rpc error: code = NotFound desc = could not find container \"b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa\": container with ID starting with b6108be701d9117618938ec9e759066c8c5fcdf03185e1c7aeda5bad853288fa not found: ID does not exist"
Nov 01 23:00:38.949532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.949505    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"]
Nov 01 23:00:38.997340 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:38.997313    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-2-w1-5b8fd2-kserve-bc4df4b68-jz7ft"]
Nov 01 23:00:39.002345 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.002323    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"]
Nov 01 23:00:39.003121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003101    4837 cpu_manager.go:414] "containerMap: removing container" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy"
Nov 01 23:00:39.003121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003119    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003132    4837 cpu_manager.go:414] "containerMap: removing container" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003137    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003144    4837 cpu_manager.go:414] "containerMap: removing container" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003149    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003161    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003167    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003189    4837 cpu_manager.go:414] "containerMap: removing container" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003194    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003206    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main"
Nov 01 23:00:39.003249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003210    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003306    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" containerName="main"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003314    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="75034634-c95f-4cb2-aa56-897dc81cef3f" containerName="cleanup"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003323    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6af8618-cbd0-4678-9dfc-088e8337ed08" containerName="cleanup"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003333    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003339    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" containerName="main"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003343    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" containerName="main"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003348    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabd9a7a-987c-47b2-9138-49889b712d21" containerName="cleanup"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003354    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a841a30-8ea9-449e-bda1-1309b98b5141" containerName="cleanup"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003360    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5787d4-6244-4b0f-a2ac-84044b041586" containerName="cleanup"
Nov 01 23:00:39.003542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.003368    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0172cd03-fd27-4b92-920b-ba2ee1b89f90" containerName="cleanup"
Nov 01 23:00:39.008907 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.008887    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.059775 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.059742    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"]
Nov 01 23:00:39.060813 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.060772    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-aa47d8-openshift-default-dockercfg-pgwd5\""
Nov 01 23:00:39.065575 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.065550    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-data\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.065695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.065585    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-podinfo\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.065695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.065619    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-envoy\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.065695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.065686    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-socket\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.065828 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.065727    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-token\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.065828 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.065805    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-certs\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.065959 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.065937    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-credential-socket\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.066149 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.066128    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istiod-ca-cert\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.066257 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.066178    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svlfl\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-kube-api-access-svlfl\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.113026 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.112963    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"]
Nov 01 23:00:39.173065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.154104    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/simulator-w1-5b8fd2-kserve-594f4cb6d9-w84nl"]
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.174430    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svlfl\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-kube-api-access-svlfl\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.174546    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-data\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.174592    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-podinfo\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.174629    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-envoy\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.174665    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-socket\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.174703    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-token\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.174824    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-certs\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.174891    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-credential-socket\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.175078    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istiod-ca-cert\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.176070    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istiod-ca-cert\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.176643    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-socket\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.176906    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-certs\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.182188    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-data\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.188069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.182775    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-credential-socket\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.263166 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.262482    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-envoy\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.263166 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.263075    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-podinfo\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.265897 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.265864    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svlfl\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-kube-api-access-svlfl\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.266698 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.266620    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-token\") pod \"e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") " pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.284166 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.277032    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"]
Nov 01 23:00:39.286271 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.286204    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/premium-w1-5b8fd2-kserve-75bd8644dd-qbbhc"]
Nov 01 23:00:39.325782 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.325753    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:39.594938 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.594912    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 23:00:39.673744 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.673713    4837 scope.go:122] "RemoveContainer" containerID="eae58a75f219598fe97aa7a7b1367c424992baa8148da0bd08bd2da9e4d154cb"
Nov 01 23:00:39.695886 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.695858    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-data\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.696019 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.695904    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-kube-api-access-48v9q\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-kube-api-access-48v9q\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.696019 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.695985    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-certs\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.696156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696021    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-token\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-token\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.696156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696104    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istiod-ca-cert\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.696263 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696181    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-credential-socket\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.696263 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696186    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-data" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.696438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696372    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-credential-socket" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.696438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696383    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-envoy\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.696623 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696558    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-podinfo\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.697079 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696775    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-certs" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.697079 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.696824    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-socket\") pod \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\" (UID: \"5324fb8a-e6de-48cb-88e7-db3d2280ff12\") "
Nov 01 23:00:39.698032 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.697964    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.698199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.698178    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.698312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.698288    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.698525 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.698504    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-socket" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.699242 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.699204    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istiod-ca-cert" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:39.699502 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.699470    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-kube-api-access-48v9q" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "kube-api-access-48v9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:39.701196 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.700734    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-token" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:39.701196 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.700819    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-envoy" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.701411 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.701384    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-podinfo" pod "5324fb8a-e6de-48cb-88e7-db3d2280ff12" (UID: "5324fb8a-e6de-48cb-88e7-db3d2280ff12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:39.746106 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.746079    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"]
Nov 01 23:00:39.756920 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:39.756896    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2f5e83_f48d_422e_a12c_65942e5ceb5f.slice/crio-d16c55bae2b1ca0d951874225b75d146fde02a8d7cb9020459e67237befae01b WatchSource:0}: Error finding container d16c55bae2b1ca0d951874225b75d146fde02a8d7cb9020459e67237befae01b: Status 404 returned error can't find the container with id d16c55bae2b1ca0d951874225b75d146fde02a8d7cb9020459e67237befae01b
Nov 01 23:00:39.762938 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.762907    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:00:39.762999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.762975    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:00:39.763032 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.763004    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:00:39.799609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.799581    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.799721 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.799616    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48v9q\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-kube-api-access-48v9q\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.799721 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.799632    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.799721 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.799645    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.799721 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.799658    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.799721 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.799671    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5324fb8a-e6de-48cb-88e7-db3d2280ff12-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.846130 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.846097    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" event={"ID":"5324fb8a-e6de-48cb-88e7-db3d2280ff12","Type":"ContainerDied","Data":"eae58a75f219598fe97aa7a7b1367c424992baa8148da0bd08bd2da9e4d154cb"}
Nov 01 23:00:39.846492 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.846166    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x" event={"ID":"5324fb8a-e6de-48cb-88e7-db3d2280ff12","Type":"ContainerDied","Data":"1835f45ddf5d4a2402c857f561d8745d05fdefc2e539210b23442a74465a49f7"}
Nov 01 23:00:39.847740 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.847717    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" event={"ID":"2b2f5e83-f48d-422e-a12c-65942e5ceb5f","Type":"ContainerStarted","Data":"d16c55bae2b1ca0d951874225b75d146fde02a8d7cb9020459e67237befae01b"}
Nov 01 23:00:39.847740 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:39.847739    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"
Nov 01 23:00:40.160626 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.160532    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"]
Nov 01 23:00:40.281515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.281424    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-7tq2x"]
Nov 01 23:00:40.648018 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.647981    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" path="/var/lib/kubelet/pods/5324fb8a-e6de-48cb-88e7-db3d2280ff12/volumes"
Nov 01 23:00:40.648928 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.648909    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b" path="/var/lib/kubelet/pods/6c7d43c7-c0d4-47a5-80c9-6b1bcc91345b/volumes"
Nov 01 23:00:40.649702 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.649658    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85be5c4e-2135-40a8-85eb-43a9021b30a2" path="/var/lib/kubelet/pods/85be5c4e-2135-40a8-85eb-43a9021b30a2/volumes"
Nov 01 23:00:40.650589 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.650571    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5890029-fa50-49f7-bf3f-0803dd176bcb" path="/var/lib/kubelet/pods/e5890029-fa50-49f7-bf3f-0803dd176bcb/volumes"
Nov 01 23:00:40.756285 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.756250    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd" podUID="485db561-5b7b-4802-88c2-31ac2dce2e51" containerName="payload-processing" containerID="cri-o://c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7" gracePeriod=25
Nov 01 23:00:40.857243 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.857210    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" event={"ID":"2b2f5e83-f48d-422e-a12c-65942e5ceb5f","Type":"ContainerStarted","Data":"480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df"}
Nov 01 23:00:40.972680 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:40.972629    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" podStartSLOduration=2.972613805 podStartE2EDuration="2.972613805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:40.968523128 +0000 UTC m=+2222.557722210" watchObservedRunningTime="2026-11-01 23:00:40.972613805 +0000 UTC m=+2222.561812904"
Nov 01 23:00:41.065414 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.065389    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 23:00:41.128063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.121350    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485db561-5b7b-4802-88c2-31ac2dce2e51-kube-api-access-dgbwb\" (UniqueName: \"kubernetes.io/projected/485db561-5b7b-4802-88c2-31ac2dce2e51-kube-api-access-dgbwb\") pod \"485db561-5b7b-4802-88c2-31ac2dce2e51\" (UID: \"485db561-5b7b-4802-88c2-31ac2dce2e51\") "
Nov 01 23:00:41.128063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.121538    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485db561-5b7b-4802-88c2-31ac2dce2e51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485db561-5b7b-4802-88c2-31ac2dce2e51-plugins-config-volume\") pod \"485db561-5b7b-4802-88c2-31ac2dce2e51\" (UID: \"485db561-5b7b-4802-88c2-31ac2dce2e51\") "
Nov 01 23:00:41.128063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.125505    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485db561-5b7b-4802-88c2-31ac2dce2e51-plugins-config-volume" pod "485db561-5b7b-4802-88c2-31ac2dce2e51" (UID: "485db561-5b7b-4802-88c2-31ac2dce2e51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:41.151921 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.151616    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485db561-5b7b-4802-88c2-31ac2dce2e51-kube-api-access-dgbwb" pod "485db561-5b7b-4802-88c2-31ac2dce2e51" (UID: "485db561-5b7b-4802-88c2-31ac2dce2e51"). InnerVolumeSpecName "kube-api-access-dgbwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:41.229175 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.229095    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485db561-5b7b-4802-88c2-31ac2dce2e51-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.229175 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.229132    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgbwb\" (UniqueName: \"kubernetes.io/projected/485db561-5b7b-4802-88c2-31ac2dce2e51-kube-api-access-dgbwb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.326497 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.326467    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:41.332351 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.332328    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:41.865555 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.865521    4837 generic.go:356] "Generic (PLEG): container finished" podID="485db561-5b7b-4802-88c2-31ac2dce2e51" containerID="c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7" exitCode=0
Nov 01 23:00:41.866059 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.865557    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd" event={"ID":"485db561-5b7b-4802-88c2-31ac2dce2e51","Type":"ContainerDied","Data":"c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7"}
Nov 01 23:00:41.866059 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.865600    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"
Nov 01 23:00:41.866059 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.865619    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd" event={"ID":"485db561-5b7b-4802-88c2-31ac2dce2e51","Type":"ContainerDied","Data":"a31d6d8946717237788e01817dd0f1c1a591b2ba6d13500ffbd8df4fbce8d793"}
Nov 01 23:00:41.866059 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.865651    4837 scope.go:122] "RemoveContainer" containerID="c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7"
Nov 01 23:00:41.866418 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.866399    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:41.867399 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.867381    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:00:41.876355 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.876206    4837 scope.go:122] "RemoveContainer" containerID="c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7"
Nov 01 23:00:41.876516 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:41.876495    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7\": container with ID starting with c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7 not found: ID does not exist" containerID="c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7"
Nov 01 23:00:41.876576 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:41.876524    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7"} err="failed to get container status \"c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7\": rpc error: code = NotFound desc = could not find container \"c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7\": container with ID starting with c7319af10155757401b6e8cf5b38120b4b03f0a82ec19ed3b143710e295e5bd7 not found: ID does not exist"
Nov 01 23:00:42.130825 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:42.130716    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"]
Nov 01 23:00:42.142453 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:42.142417    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-103369-77fd8cdbb7-z27pd"]
Nov 01 23:00:42.527558 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:42.527524    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485db561-5b7b-4802-88c2-31ac2dce2e51" path="/var/lib/kubelet/pods/485db561-5b7b-4802-88c2-31ac2dce2e51/volumes"
Nov 01 23:00:48.535225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:48.535188    4837 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod8c17bfba-c48b-41de-872a-c491be361d55"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod8c17bfba-c48b-41de-872a-c491be361d55] : Timed out while waiting for systemd to remove kubepods-burstable-pod8c17bfba_c48b_41de_872a_c491be361d55.slice"
Nov 01 23:00:48.535225 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:48.535226    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod8c17bfba-c48b-41de-872a-c491be361d55] : unable to destroy cgroup paths for cgroup [kubepods burstable pod8c17bfba-c48b-41de-872a-c491be361d55] : Timed out while waiting for systemd to remove kubepods-burstable-pod8c17bfba_c48b_41de_872a_c491be361d55.slice" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9" podUID="8c17bfba-c48b-41de-872a-c491be361d55"
Nov 01 23:00:48.814247 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:00:48.814162    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:00:48.914067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:48.914025    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"
Nov 01 23:00:49.125699 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:49.125612    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"]
Nov 01 23:00:49.132185 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:49.132158    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892900-xzfg9"]
Nov 01 23:00:50.159104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.157938    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"]
Nov 01 23:00:50.159104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.158975    4837 cpu_manager.go:414] "containerMap: removing container" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy"
Nov 01 23:00:50.159104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.158995    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy"
Nov 01 23:00:50.183068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.180124    4837 cpu_manager.go:414] "containerMap: removing container" podUID="485db561-5b7b-4802-88c2-31ac2dce2e51" containerName="payload-processing"
Nov 01 23:00:50.183068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.180158    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485db561-5b7b-4802-88c2-31ac2dce2e51" containerName="payload-processing"
Nov 01 23:00:50.183068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.180200    4837 cpu_manager.go:414] "containerMap: removing container" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy"
Nov 01 23:00:50.183068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.180208    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy"
Nov 01 23:00:50.183068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.180484    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="485db561-5b7b-4802-88c2-31ac2dce2e51" containerName="payload-processing"
Nov 01 23:00:50.183068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.180504    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5324fb8a-e6de-48cb-88e7-db3d2280ff12" containerName="istio-proxy"
Nov 01 23:00:50.183068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.180672    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e21ad7-e679-4381-a949-814c8b5207cd" containerName="istio-proxy"
Nov 01 23:00:50.195310 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.193148    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:50.251062 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.237342    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"]
Nov 01 23:00:50.273066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.271478    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-aa47d8-dockercfg-h7nfl\""
Nov 01 23:00:50.273066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.272181    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-aa47d8\""
Nov 01 23:00:50.339164 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.339122    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55z7n\" (UniqueName: \"kubernetes.io/projected/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-kube-api-access-55z7n\") pod \"payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk\" (UID: \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:50.339892 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.339459    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk\" (UID: \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:50.440761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.440729    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk\" (UID: \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:50.440939 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.440857    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55z7n\" (UniqueName: \"kubernetes.io/projected/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-kube-api-access-55z7n\") pod \"payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk\" (UID: \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:50.441549 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.441531    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk\" (UID: \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:50.494116 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.494087    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55z7n\" (UniqueName: \"kubernetes.io/projected/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-kube-api-access-55z7n\") pod \"payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk\" (UID: \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:50.572883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.572850    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:50.583600 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.583553    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c17bfba-c48b-41de-872a-c491be361d55" path="/var/lib/kubelet/pods/8c17bfba-c48b-41de-872a-c491be361d55/volumes"
Nov 01 23:00:50.955236 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:00:50.955191    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab1b3f3_5281_4815_87cd_df2dbea8b90c.slice/crio-3d51709dce3c0934f8e1f0d01ab94c1e535bd6424b4accdac218fed95c05305f WatchSource:0}: Error finding container 3d51709dce3c0934f8e1f0d01ab94c1e535bd6424b4accdac218fed95c05305f: Status 404 returned error can't find the container with id 3d51709dce3c0934f8e1f0d01ab94c1e535bd6424b4accdac218fed95c05305f
Nov 01 23:00:50.955424 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:50.955352    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"]
Nov 01 23:00:51.964898 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:51.964862    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk" event={"ID":"0ab1b3f3-5281-4815-87cd-df2dbea8b90c","Type":"ContainerStarted","Data":"f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0"}
Nov 01 23:00:51.965315 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:51.964913    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk" event={"ID":"0ab1b3f3-5281-4815-87cd-df2dbea8b90c","Type":"ContainerStarted","Data":"3d51709dce3c0934f8e1f0d01ab94c1e535bd6424b4accdac218fed95c05305f"}
Nov 01 23:00:51.965315 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:51.964952    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:00:52.075106 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:00:52.075016    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk" podStartSLOduration=2.075000328 podStartE2EDuration="2.075000328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:52.071621995 +0000 UTC m=+2233.660821057" watchObservedRunningTime="2026-11-01 23:00:52.075000328 +0000 UTC m=+2233.664199393"
Nov 01 23:01:02.975038 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:02.975003    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:01:10.701919 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:10.701881    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"]
Nov 01 23:01:10.702526 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:10.702148    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq" podUID="7fbadf4f-8515-48eb-bff3-6a993d35e3a4" containerName="maas-api" containerID="cri-o://abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1" gracePeriod=30
Nov 01 23:01:11.062455 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.062428    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 23:01:11.188690 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.188654    4837 generic.go:356] "Generic (PLEG): container finished" podID="7fbadf4f-8515-48eb-bff3-6a993d35e3a4" containerID="abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1" exitCode=0
Nov 01 23:01:11.189307 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.189154    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"
Nov 01 23:01:11.189794 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.189192    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq" event={"ID":"7fbadf4f-8515-48eb-bff3-6a993d35e3a4","Type":"ContainerDied","Data":"abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1"}
Nov 01 23:01:11.189964 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.189948    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq" event={"ID":"7fbadf4f-8515-48eb-bff3-6a993d35e3a4","Type":"ContainerDied","Data":"e54c247a197b0d24cade06c6d9e882b501c8f9f0679197173bc6b22390714a9b"}
Nov 01 23:01:11.190213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.190197    4837 scope.go:122] "RemoveContainer" containerID="abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1"
Nov 01 23:01:11.191058 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.191025    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-metrics-certs\") pod \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") "
Nov 01 23:01:11.191673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.191650    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-openshift-service-ca\") pod \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") "
Nov 01 23:01:11.192491 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.192468    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-kube-api-access-44tm6\" (UniqueName: \"kubernetes.io/projected/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-kube-api-access-44tm6\") pod \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") "
Nov 01 23:01:11.193128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.193106    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls\") pod \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\" (UID: \"7fbadf4f-8515-48eb-bff3-6a993d35e3a4\") "
Nov 01 23:01:11.196357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.192169    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-openshift-service-ca" pod "7fbadf4f-8515-48eb-bff3-6a993d35e3a4" (UID: "7fbadf4f-8515-48eb-bff3-6a993d35e3a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:11.196883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.196775    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-metrics-certs" pod "7fbadf4f-8515-48eb-bff3-6a993d35e3a4" (UID: "7fbadf4f-8515-48eb-bff3-6a993d35e3a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:01:11.207745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.207709    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls" pod "7fbadf4f-8515-48eb-bff3-6a993d35e3a4" (UID: "7fbadf4f-8515-48eb-bff3-6a993d35e3a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:01:11.211687 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.211644    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-kube-api-access-44tm6" pod "7fbadf4f-8515-48eb-bff3-6a993d35e3a4" (UID: "7fbadf4f-8515-48eb-bff3-6a993d35e3a4"). InnerVolumeSpecName "kube-api-access-44tm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:11.243977 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.243946    4837 scope.go:122] "RemoveContainer" containerID="abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1"
Nov 01 23:01:11.246139 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:01:11.246105    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1\": container with ID starting with abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1 not found: ID does not exist" containerID="abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1"
Nov 01 23:01:11.246307 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.246279    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1"} err="failed to get container status \"abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1\": rpc error: code = NotFound desc = could not find container \"abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1\": container with ID starting with abbe25934966b75b4eca91baaba8106636c0d6f5aea4d9f61162582d2a7704a1 not found: ID does not exist"
Nov 01 23:01:11.295968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.295909    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:11.295968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.295934    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:11.295968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.295945    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:11.295968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.295956    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44tm6\" (UniqueName: \"kubernetes.io/projected/7fbadf4f-8515-48eb-bff3-6a993d35e3a4-kube-api-access-44tm6\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:11.605640 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.605560    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"]
Nov 01 23:01:11.691741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:11.691701    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd7ff8-789bc5d47b-jqrvq"]
Nov 01 23:01:12.528365 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:12.528330    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fbadf4f-8515-48eb-bff3-6a993d35e3a4" path="/var/lib/kubelet/pods/7fbadf4f-8515-48eb-bff3-6a993d35e3a4/volumes"
Nov 01 23:01:18.815507 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:01:18.815478    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:01:28.807833 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:28.807796    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:01:30.932397 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:30.932365    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"]
Nov 01 23:01:31.031396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:31.031365    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"]
Nov 01 23:01:35.933192 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:35.933148    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb" podUID="c493d14a-2273-48ad-9a2a-04e39b147078" containerName="payload-processing" containerID="cri-o://6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda" gracePeriod=25
Nov 01 23:01:36.031911 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.031881    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft" podUID="73f8cc97-3f57-4e2f-95eb-148260e80ef9" containerName="payload-pre-processing" containerID="cri-o://05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f" gracePeriod=25
Nov 01 23:01:36.217573 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.217546    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 23:01:36.298371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.298347    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c493d14a-2273-48ad-9a2a-04e39b147078-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493d14a-2273-48ad-9a2a-04e39b147078-plugins-config-volume\") pod \"c493d14a-2273-48ad-9a2a-04e39b147078\" (UID: \"c493d14a-2273-48ad-9a2a-04e39b147078\") "
Nov 01 23:01:36.298636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.298611    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c493d14a-2273-48ad-9a2a-04e39b147078-kube-api-access-cj2p4\" (UniqueName: \"kubernetes.io/projected/c493d14a-2273-48ad-9a2a-04e39b147078-kube-api-access-cj2p4\") pod \"c493d14a-2273-48ad-9a2a-04e39b147078\" (UID: \"c493d14a-2273-48ad-9a2a-04e39b147078\") "
Nov 01 23:01:36.298714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.298642    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c493d14a-2273-48ad-9a2a-04e39b147078-plugins-config-volume" pod "c493d14a-2273-48ad-9a2a-04e39b147078" (UID: "c493d14a-2273-48ad-9a2a-04e39b147078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:36.299186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.299164    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c493d14a-2273-48ad-9a2a-04e39b147078-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:36.300675 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.300658    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c493d14a-2273-48ad-9a2a-04e39b147078-kube-api-access-cj2p4" pod "c493d14a-2273-48ad-9a2a-04e39b147078" (UID: "c493d14a-2273-48ad-9a2a-04e39b147078"). InnerVolumeSpecName "kube-api-access-cj2p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:36.301781 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.301764    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 23:01:36.400442 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.400412    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f8cc97-3f57-4e2f-95eb-148260e80ef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8cc97-3f57-4e2f-95eb-148260e80ef9-plugins-config-volume\") pod \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\" (UID: \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\") "
Nov 01 23:01:36.400601 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.400588    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f8cc97-3f57-4e2f-95eb-148260e80ef9-kube-api-access-2pdxf\" (UniqueName: \"kubernetes.io/projected/73f8cc97-3f57-4e2f-95eb-148260e80ef9-kube-api-access-2pdxf\") pod \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\" (UID: \"73f8cc97-3f57-4e2f-95eb-148260e80ef9\") "
Nov 01 23:01:36.400764 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.400743    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f8cc97-3f57-4e2f-95eb-148260e80ef9-plugins-config-volume" pod "73f8cc97-3f57-4e2f-95eb-148260e80ef9" (UID: "73f8cc97-3f57-4e2f-95eb-148260e80ef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:36.401129 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.401096    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f8cc97-3f57-4e2f-95eb-148260e80ef9-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:36.401238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.401134    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2p4\" (UniqueName: \"kubernetes.io/projected/c493d14a-2273-48ad-9a2a-04e39b147078-kube-api-access-cj2p4\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:36.402683 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.402663    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f8cc97-3f57-4e2f-95eb-148260e80ef9-kube-api-access-2pdxf" pod "73f8cc97-3f57-4e2f-95eb-148260e80ef9" (UID: "73f8cc97-3f57-4e2f-95eb-148260e80ef9"). InnerVolumeSpecName "kube-api-access-2pdxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:36.497583 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.497512    4837 generic.go:356] "Generic (PLEG): container finished" podID="c493d14a-2273-48ad-9a2a-04e39b147078" containerID="6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda" exitCode=0
Nov 01 23:01:36.497583 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.497554    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"
Nov 01 23:01:36.497753 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.497595    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb" event={"ID":"c493d14a-2273-48ad-9a2a-04e39b147078","Type":"ContainerDied","Data":"6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda"}
Nov 01 23:01:36.497753 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.497662    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb" event={"ID":"c493d14a-2273-48ad-9a2a-04e39b147078","Type":"ContainerDied","Data":"01cd1cdbb5e597b0920c9dee6fbe6c33c68af0034f11a6ee6066b6d9cd8853ff"}
Nov 01 23:01:36.497753 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.497693    4837 scope.go:122] "RemoveContainer" containerID="6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda"
Nov 01 23:01:36.499022 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.499003    4837 generic.go:356] "Generic (PLEG): container finished" podID="73f8cc97-3f57-4e2f-95eb-148260e80ef9" containerID="05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f" exitCode=0
Nov 01 23:01:36.499135 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.499072    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft" event={"ID":"73f8cc97-3f57-4e2f-95eb-148260e80ef9","Type":"ContainerDied","Data":"05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f"}
Nov 01 23:01:36.499135 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.499082    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"
Nov 01 23:01:36.499135 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.499111    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft" event={"ID":"73f8cc97-3f57-4e2f-95eb-148260e80ef9","Type":"ContainerDied","Data":"4f0e56c61f723eb7187a6d9ac5789dc67bf3acb2acf58e410fea8f3247175e76"}
Nov 01 23:01:36.502514 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.502497    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdxf\" (UniqueName: \"kubernetes.io/projected/73f8cc97-3f57-4e2f-95eb-148260e80ef9-kube-api-access-2pdxf\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:36.508916 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.508885    4837 scope.go:122] "RemoveContainer" containerID="6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda"
Nov 01 23:01:36.509244 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:01:36.509218    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda\": container with ID starting with 6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda not found: ID does not exist" containerID="6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda"
Nov 01 23:01:36.509312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.509251    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda"} err="failed to get container status \"6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda\": rpc error: code = NotFound desc = could not find container \"6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda\": container with ID starting with 6237b2e6482374b5c294a0ebf79cfa03a07807944340151fdf3a706aa9bbebda not found: ID does not exist"
Nov 01 23:01:36.509312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.509273    4837 scope.go:122] "RemoveContainer" containerID="05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f"
Nov 01 23:01:36.518498 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.518481    4837 scope.go:122] "RemoveContainer" containerID="05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f"
Nov 01 23:01:36.530814 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:01:36.530778    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f\": container with ID starting with 05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f not found: ID does not exist" containerID="05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f"
Nov 01 23:01:36.530957 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.530819    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f"} err="failed to get container status \"05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f\": rpc error: code = NotFound desc = could not find container \"05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f\": container with ID starting with 05f156b617e67eabb8394ce1e280af2930cf4aaa168237ae791f0d85a670673f not found: ID does not exist"
Nov 01 23:01:36.652365 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.652328    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"]
Nov 01 23:01:36.656496 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.656470    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-584561-75964b6987-nm5tb"]
Nov 01 23:01:36.757774 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.757695    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"]
Nov 01 23:01:36.762250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:36.762223    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-584561-665f6fdf95-sfmft"]
Nov 01 23:01:38.527350 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:38.527318    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f8cc97-3f57-4e2f-95eb-148260e80ef9" path="/var/lib/kubelet/pods/73f8cc97-3f57-4e2f-95eb-148260e80ef9/volumes"
Nov 01 23:01:38.527922 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:38.527901    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c493d14a-2273-48ad-9a2a-04e39b147078" path="/var/lib/kubelet/pods/c493d14a-2273-48ad-9a2a-04e39b147078/volumes"
Nov 01 23:01:48.417683 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.417611    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"]
Nov 01 23:01:48.418374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418299    4837 cpu_manager.go:414] "containerMap: removing container" podUID="c493d14a-2273-48ad-9a2a-04e39b147078" containerName="payload-processing"
Nov 01 23:01:48.418374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418313    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c493d14a-2273-48ad-9a2a-04e39b147078" containerName="payload-processing"
Nov 01 23:01:48.418374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418335    4837 cpu_manager.go:414] "containerMap: removing container" podUID="7fbadf4f-8515-48eb-bff3-6a993d35e3a4" containerName="maas-api"
Nov 01 23:01:48.418374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418340    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fbadf4f-8515-48eb-bff3-6a993d35e3a4" containerName="maas-api"
Nov 01 23:01:48.418374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418355    4837 cpu_manager.go:414] "containerMap: removing container" podUID="73f8cc97-3f57-4e2f-95eb-148260e80ef9" containerName="payload-pre-processing"
Nov 01 23:01:48.418374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418361    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f8cc97-3f57-4e2f-95eb-148260e80ef9" containerName="payload-pre-processing"
Nov 01 23:01:48.418604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418459    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c493d14a-2273-48ad-9a2a-04e39b147078" containerName="payload-processing"
Nov 01 23:01:48.418604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418469    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fbadf4f-8515-48eb-bff3-6a993d35e3a4" containerName="maas-api"
Nov 01 23:01:48.418604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.418476    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f8cc97-3f57-4e2f-95eb-148260e80ef9" containerName="payload-pre-processing"
Nov 01 23:01:48.423570 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.423552    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.471826 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.471797    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-675d1d-serving-cert\""
Nov 01 23:01:48.476782 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.476760    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-openshift-service-ca\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.476880 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.476810    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.476947 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.476935    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4c45\" (UniqueName: \"kubernetes.io/projected/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-kube-api-access-l4c45\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.476989 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.476978    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-metrics-certs\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.540570 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.540541    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"]
Nov 01 23:01:48.578105 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.578069    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-metrics-certs\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.578298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.578282    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-openshift-service-ca\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.578377 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.578340    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.578542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.578521    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4c45\" (UniqueName: \"kubernetes.io/projected/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-kube-api-access-l4c45\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.578743 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:01:48.578722    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-serving-cert: secret "maas-api-e2e-inf-a-675d1d-serving-cert" not found
Nov 01 23:01:48.578828 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:01:48.578788    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls podName:9358a5c7-3313-4dee-b5e4-d0ac7a810fde nodeName:}" failed. No retries permitted until 2026-11-01 23:01:49.07876925 +0000 UTC m=+2290.667968313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls") pod "maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b" (UID: "9358a5c7-3313-4dee-b5e4-d0ac7a810fde") : secret "maas-api-e2e-inf-a-675d1d-serving-cert" not found
Nov 01 23:01:48.579810 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.579780    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-openshift-service-ca\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.582035 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.582007    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-metrics-certs\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.632889 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:48.632867    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4c45\" (UniqueName: \"kubernetes.io/projected/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-kube-api-access-l4c45\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:48.816124 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:01:48.816008    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:01:49.092495 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:49.092411    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:49.094861 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:49.094838    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls\") pod \"maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:49.348233 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:49.348163    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:49.765569 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:49.765503    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"]
Nov 01 23:01:49.770089 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:01:49.770060    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9358a5c7_3313_4dee_b5e4_d0ac7a810fde.slice/crio-a039ea9aeb00a3b93b7aa77f88ce7ad4ad3813a15e5f2bf67fd06d6b744bb571 WatchSource:0}: Error finding container a039ea9aeb00a3b93b7aa77f88ce7ad4ad3813a15e5f2bf67fd06d6b744bb571: Status 404 returned error can't find the container with id a039ea9aeb00a3b93b7aa77f88ce7ad4ad3813a15e5f2bf67fd06d6b744bb571
Nov 01 23:01:49.771966 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:49.771946    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 23:01:50.602723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:50.602687    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b" event={"ID":"9358a5c7-3313-4dee-b5e4-d0ac7a810fde","Type":"ContainerStarted","Data":"a039ea9aeb00a3b93b7aa77f88ce7ad4ad3813a15e5f2bf67fd06d6b744bb571"}
Nov 01 23:01:51.610517 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:51.610480    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b" event={"ID":"9358a5c7-3313-4dee-b5e4-d0ac7a810fde","Type":"ContainerStarted","Data":"c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86"}
Nov 01 23:01:51.611009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:51.610540    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:01:51.720159 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:51.720106    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b" podStartSLOduration=2.842440091 podStartE2EDuration="3.720090098s" totalImagesPullingTime="877.650007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:51.717700657 +0000 UTC m=+2293.306899718" watchObservedRunningTime="2026-11-01 23:01:51.720090098 +0000 UTC m=+2293.309289162"
Nov 01 23:01:55.571269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:55.571235    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"]
Nov 01 23:01:55.571676 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:55.571455    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy" containerID="cri-o://3ed33fbbe2633d7da985d8f65f031b6fa0968c8e6963e57cd399d6ad3dd4a37f" gracePeriod=30
Nov 01 23:01:57.621605 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:01:57.621576    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:02:00.711837 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:00.711799    4837 generic.go:356] "Generic (PLEG): container finished" podID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerID="3ed33fbbe2633d7da985d8f65f031b6fa0968c8e6963e57cd399d6ad3dd4a37f" exitCode=0
Nov 01 23:02:00.712246 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:00.711867    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" event={"ID":"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad","Type":"ContainerDied","Data":"3ed33fbbe2633d7da985d8f65f031b6fa0968c8e6963e57cd399d6ad3dd4a37f"}
Nov 01 23:02:00.712246 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:00.711929    4837 scope.go:122] "RemoveContainer" containerID="23bbea1a85e4fbb2e552da906c2e7d9dac078467d5ca1a75b8ee3cbf1ddb3825"
Nov 01 23:02:00.855760 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:00.855705    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 23:02:01.051108 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051072    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istiod-ca-cert\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051236    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-envoy\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051276    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-certs\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051445 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051331    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-token\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-token\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051445 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051369    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-kube-api-access-dvvhx\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-kube-api-access-dvvhx\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051445 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051404    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-podinfo\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051445 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051440    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-credential-socket\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051453    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istiod-ca-cert" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:01.051673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051543    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-socket\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051589    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-certs" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:01.051835 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051705    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-data\") pod \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\" (UID: \"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad\") "
Nov 01 23:02:01.051896 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.051835    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-socket" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:01.052139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.052115    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-data" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:01.052320 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.052259    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-credential-socket" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:01.052579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.052551    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.052579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.052576    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.052766 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.052588    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.052766 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.052597    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.052766 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.052606    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.054200 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.054174    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-podinfo" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:02:01.054558 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.054529    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-envoy" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:01.054736 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.054718    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-kube-api-access-dvvhx" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "kube-api-access-dvvhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:01.054785 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.054769    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-token" pod "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" (UID: "ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:01.153639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.153576    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.153639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.153600    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.153639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.153611    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvvhx\" (UniqueName: \"kubernetes.io/projected/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-kube-api-access-dvvhx\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.153639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.153621    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:01.721425 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.721385    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"
Nov 01 23:02:01.721848 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.721429    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj" event={"ID":"ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad","Type":"ContainerDied","Data":"5fc89396c4cdaaada90b6f6c03fe9b58fcd78e8b917dc6a85f212b7febefdec1"}
Nov 01 23:02:01.721848 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.721488    4837 scope.go:122] "RemoveContainer" containerID="3ed33fbbe2633d7da985d8f65f031b6fa0968c8e6963e57cd399d6ad3dd4a37f"
Nov 01 23:02:01.923630 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.923594    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"]
Nov 01 23:02:01.970193 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:01.970164    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-103369-openshift-default-86d5c6bd9-c7dzj"]
Nov 01 23:02:02.526780 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:02.526745    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" path="/var/lib/kubelet/pods/ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad/volumes"
Nov 01 23:02:04.052627 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.052591    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"]
Nov 01 23:02:04.223460 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.223428    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"]
Nov 01 23:02:04.224224 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.224203    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy"
Nov 01 23:02:04.224224 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.224221    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy"
Nov 01 23:02:04.224381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.224249    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy"
Nov 01 23:02:04.224381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.224254    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy"
Nov 01 23:02:04.224381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.224361    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy"
Nov 01 23:02:04.224381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.224374    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae55be6e-3c9e-450e-8ca6-fbb0a0a48dad" containerName="istio-proxy"
Nov 01 23:02:04.229197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.229176    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.279468 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.279440    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-aa47d8-serving-cert\""
Nov 01 23:02:04.284117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.284096    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"]
Nov 01 23:02:04.293435 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.293407    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-metrics-certs\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.293538 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.293480    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17dc971-5864-4d0a-807e-78e593e3fec1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.293719 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.293667    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhk9n\" (UniqueName: \"kubernetes.io/projected/e17dc971-5864-4d0a-807e-78e593e3fec1-kube-api-access-rhk9n\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.293837 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.293737    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.395270 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.395185    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-metrics-certs\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.395270 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.395262    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17dc971-5864-4d0a-807e-78e593e3fec1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.395499 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.395378    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhk9n\" (UniqueName: \"kubernetes.io/projected/e17dc971-5864-4d0a-807e-78e593e3fec1-kube-api-access-rhk9n\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.395499 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.395415    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.395626 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:02:04.395604    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-serving-cert: secret "maas-api-e2e-inf-a-aa47d8-serving-cert" not found
Nov 01 23:02:04.395704 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:02:04.395691    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls podName:e17dc971-5864-4d0a-807e-78e593e3fec1 nodeName:}" failed. No retries permitted until 2026-11-01 23:02:04.895670348 +0000 UTC m=+2306.484869402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls") pod "maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j" (UID: "e17dc971-5864-4d0a-807e-78e593e3fec1") : secret "maas-api-e2e-inf-a-aa47d8-serving-cert" not found
Nov 01 23:02:04.396055 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.396028    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17dc971-5864-4d0a-807e-78e593e3fec1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.397587 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.397570    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-metrics-certs\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.452093 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.452066    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhk9n\" (UniqueName: \"kubernetes.io/projected/e17dc971-5864-4d0a-807e-78e593e3fec1-kube-api-access-rhk9n\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.914304 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.914257    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:04.968103 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:04.966988    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:05.150221 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:05.150189    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:05.576737 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:05.576709    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"]
Nov 01 23:02:05.752478 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:05.752445    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j" event={"ID":"e17dc971-5864-4d0a-807e-78e593e3fec1","Type":"ContainerStarted","Data":"4099c41bdf8e5c66d62d83a9a66693ac5f44c165234d74d8d5d8deb4c1f523de"}
Nov 01 23:02:06.761616 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:06.761571    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j" event={"ID":"e17dc971-5864-4d0a-807e-78e593e3fec1","Type":"ContainerStarted","Data":"3afc0e51525957d3c0619ff56de993bfb562e55081d17c09f0a8babe87ba135f"}
Nov 01 23:02:06.762130 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:06.761660    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:06.872636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:06.872581    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j" podStartSLOduration=2.036338744 podStartE2EDuration="2.872565293s" totalImagesPullingTime="836.226549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:06.871032364 +0000 UTC m=+2308.460231428" watchObservedRunningTime="2026-11-01 23:02:06.872565293 +0000 UTC m=+2308.461764355"
Nov 01 23:02:11.515723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:11.515689    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"]
Nov 01 23:02:11.561641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:11.561609    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5b8fd2-hgv7z"]
Nov 01 23:02:12.543497 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:12.543457    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3269d234-2352-4493-819d-007ffca37155" path="/var/lib/kubelet/pods/3269d234-2352-4493-819d-007ffca37155/volumes"
Nov 01 23:02:12.773945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:12.773917    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:02:14.823467 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:14.823435    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:02:17.025030 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:17.024983    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:02:17.530646 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:17.530593    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:02:17.797710 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:17.797611    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Nov 01 23:02:17.887390 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:17.887129    4837 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:02:18.013274 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.013223    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:02:18.029333 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.029296    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:02:18.030850 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.030829    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 23:02:18.057379 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.057290    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:02:18.320498 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.319763    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 23:02:18.500177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.500142    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"]
Nov 01 23:02:18.500682 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.500649    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy" containerID="cri-o://e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995" gracePeriod=30
Nov 01 23:02:18.608625 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.608541    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"]
Nov 01 23:02:18.609536 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:18.609471    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy" containerID="cri-o://cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc" gracePeriod=30
Nov 01 23:02:18.948278 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:02:18.948236    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:02:19.074092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.074055    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main" containerID="cri-o://81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24" gracePeriod=45
Nov 01 23:02:19.388806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.388775    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 23:02:19.486354 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.486326    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-dshm\") pod \"9e389acf-e06f-4432-bf17-2d302059ef43\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") "
Nov 01 23:02:19.486526 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.486508    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e389acf-e06f-4432-bf17-2d302059ef43-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e389acf-e06f-4432-bf17-2d302059ef43-tls-certs\") pod \"9e389acf-e06f-4432-bf17-2d302059ef43\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") "
Nov 01 23:02:19.486618 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.486603    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-tmp-dir\") pod \"9e389acf-e06f-4432-bf17-2d302059ef43\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") "
Nov 01 23:02:19.486687 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.486669    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-model-cache\") pod \"9e389acf-e06f-4432-bf17-2d302059ef43\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") "
Nov 01 23:02:19.486752 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.486706    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e389acf-e06f-4432-bf17-2d302059ef43-kube-api-access-cd7qx\" (UniqueName: \"kubernetes.io/projected/9e389acf-e06f-4432-bf17-2d302059ef43-kube-api-access-cd7qx\") pod \"9e389acf-e06f-4432-bf17-2d302059ef43\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") "
Nov 01 23:02:19.486857 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.486836    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-home\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-home\") pod \"9e389acf-e06f-4432-bf17-2d302059ef43\" (UID: \"9e389acf-e06f-4432-bf17-2d302059ef43\") "
Nov 01 23:02:19.487770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.486930    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-tmp-dir" pod "9e389acf-e06f-4432-bf17-2d302059ef43" (UID: "9e389acf-e06f-4432-bf17-2d302059ef43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.487770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.486986    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-model-cache" pod "9e389acf-e06f-4432-bf17-2d302059ef43" (UID: "9e389acf-e06f-4432-bf17-2d302059ef43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.487770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.487205    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-home" pod "9e389acf-e06f-4432-bf17-2d302059ef43" (UID: "9e389acf-e06f-4432-bf17-2d302059ef43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.487770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.487693    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.487770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.487709    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.487770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.487718    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.489257 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.489194    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-dshm" pod "9e389acf-e06f-4432-bf17-2d302059ef43" (UID: "9e389acf-e06f-4432-bf17-2d302059ef43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.489505 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.489474    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e389acf-e06f-4432-bf17-2d302059ef43-kube-api-access-cd7qx" pod "9e389acf-e06f-4432-bf17-2d302059ef43" (UID: "9e389acf-e06f-4432-bf17-2d302059ef43"). InnerVolumeSpecName "kube-api-access-cd7qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:19.489586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.489571    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e389acf-e06f-4432-bf17-2d302059ef43-tls-certs" pod "9e389acf-e06f-4432-bf17-2d302059ef43" (UID: "9e389acf-e06f-4432-bf17-2d302059ef43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:19.589247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.589171    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e389acf-e06f-4432-bf17-2d302059ef43-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.589247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.589202    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd7qx\" (UniqueName: \"kubernetes.io/projected/9e389acf-e06f-4432-bf17-2d302059ef43-kube-api-access-cd7qx\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.589247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:19.589213    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e389acf-e06f-4432-bf17-2d302059ef43-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:20.087084 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.087028    4837 generic.go:356] "Generic (PLEG): container finished" podID="9e389acf-e06f-4432-bf17-2d302059ef43" containerID="81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24" exitCode=0
Nov 01 23:02:20.087532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.087098    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"
Nov 01 23:02:20.087532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.087111    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" event={"ID":"9e389acf-e06f-4432-bf17-2d302059ef43","Type":"ContainerDied","Data":"81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24"}
Nov 01 23:02:20.087532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.087163    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p" event={"ID":"9e389acf-e06f-4432-bf17-2d302059ef43","Type":"ContainerDied","Data":"eec9f7fb1b5fff1e967e171c3fa6960df3646802a4f41ebbd71f7816510fc1c3"}
Nov 01 23:02:20.087532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.087182    4837 scope.go:122] "RemoveContainer" containerID="81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24"
Nov 01 23:02:20.098345 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.098322    4837 scope.go:122] "RemoveContainer" containerID="81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24"
Nov 01 23:02:20.098637 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:02:20.098620    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24\": container with ID starting with 81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24 not found: ID does not exist" containerID="81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24"
Nov 01 23:02:20.098707 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.098645    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24"} err="failed to get container status \"81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24\": rpc error: code = NotFound desc = could not find container \"81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24\": container with ID starting with 81e1221d276d617e5cbcd688636564d0b76b3f7d73c30b1c0273c27038166c24 not found: ID does not exist"
Nov 01 23:02:20.258018 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.257979    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"]
Nov 01 23:02:20.300150 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.300118    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-2-w5-d6101b-kserve-5cdd74bcf5-zjg4p"]
Nov 01 23:02:20.528166 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:20.528135    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" path="/var/lib/kubelet/pods/9e389acf-e06f-4432-bf17-2d302059ef43/volumes"
Nov 01 23:02:22.723775 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:22.723734    4837 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:02:22.724181 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:22.723802    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:02:23.741266 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.741241    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 23:02:23.844976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.844938    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-credential-socket\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.845199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845020    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-socket\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.845199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845137    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-kube-api-access-qzlvk\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-kube-api-access-qzlvk\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.845199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845160    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-podinfo\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.845199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845176    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-token\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.845441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845220    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-data\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.845441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845246    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istiod-ca-cert\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.845441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845382    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-certs\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.845441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845403    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-envoy\") pod \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\" (UID: \"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95\") "
Nov 01 23:02:23.846332 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.845957    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-data" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:23.846332 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.846255    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-credential-socket" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:23.846332 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.846297    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istiod-ca-cert" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:23.846677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.846490    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-socket" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:23.846677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.846578    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-certs" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:23.848550 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.848511    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-kube-api-access-qzlvk" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "kube-api-access-qzlvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:23.849482 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.849447    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-envoy" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:23.849727 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.849692    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-podinfo" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:02:23.852121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.852084    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-token" pod "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" (UID: "28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:23.905640 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.905617    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 23:02:23.947911 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.947867    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:23.947911 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.947911    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:23.948213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.947926    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:23.948213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.947939    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:23.948213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.947953    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:23.948213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.947971    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:23.948213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.947984    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:23.948213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.947998    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:23.948213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:23.948011    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzlvk\" (UniqueName: \"kubernetes.io/projected/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95-kube-api-access-qzlvk\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.050628 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.049831    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-token\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-token\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.051395 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.051220    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-credential-socket\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.051395 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.051286    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-envoy\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.051738 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.051642    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-kube-api-access-xs9k2\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-kube-api-access-xs9k2\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.051738 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.051695    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-certs\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.052208 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.052189    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-data\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.052374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.052308    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-podinfo\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.052477 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.052354    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-socket\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.052945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.052681    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e14863a-1388-4ccd-8c62-8630cc3ba882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e14863a-1388-4ccd-8c62-8630cc3ba882-istiod-ca-cert\") pod \"8e14863a-1388-4ccd-8c62-8630cc3ba882\" (UID: \"8e14863a-1388-4ccd-8c62-8630cc3ba882\") "
Nov 01 23:02:24.054603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.053593    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-certs" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.054603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.053812    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-credential-socket" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.054603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.054139    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-data" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.054603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.054278    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.054603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.054304    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.054603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.054318    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.055115 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.055080    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-token" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:24.055234 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.055157    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-socket" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.055234 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.055218    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e14863a-1388-4ccd-8c62-8630cc3ba882-istiod-ca-cert" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:24.059436 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.059407    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-podinfo" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:02:24.059552 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.059413    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-envoy" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.060633 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.060604    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-kube-api-access-xs9k2" pod "8e14863a-1388-4ccd-8c62-8630cc3ba882" (UID: "8e14863a-1388-4ccd-8c62-8630cc3ba882"). InnerVolumeSpecName "kube-api-access-xs9k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:24.117406 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.117372    4837 generic.go:356] "Generic (PLEG): container finished" podID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerID="e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995" exitCode=0
Nov 01 23:02:24.117566 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.117418    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"
Nov 01 23:02:24.117566 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.117453    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" event={"ID":"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95","Type":"ContainerDied","Data":"e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995"}
Nov 01 23:02:24.117566 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.117510    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh" event={"ID":"28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95","Type":"ContainerDied","Data":"c305ce0f76c12df418dea2c25a6ac78414f3ef9d9455d9cce9787472d1f16415"}
Nov 01 23:02:24.117566 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.117540    4837 scope.go:122] "RemoveContainer" containerID="e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995"
Nov 01 23:02:24.119192 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.119167    4837 generic.go:356] "Generic (PLEG): container finished" podID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerID="cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc" exitCode=0
Nov 01 23:02:24.119324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.119224    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"
Nov 01 23:02:24.119324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.119293    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" event={"ID":"8e14863a-1388-4ccd-8c62-8630cc3ba882","Type":"ContainerDied","Data":"cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc"}
Nov 01 23:02:24.119442 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.119334    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9" event={"ID":"8e14863a-1388-4ccd-8c62-8630cc3ba882","Type":"ContainerDied","Data":"4873829735a0556bacbf68cb668a43a7f565ba1ede91dacf1b820e6988c0b249"}
Nov 01 23:02:24.129739 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.129721    4837 scope.go:122] "RemoveContainer" containerID="e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995"
Nov 01 23:02:24.130082 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:02:24.129987    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995\": container with ID starting with e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995 not found: ID does not exist" containerID="e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995"
Nov 01 23:02:24.130082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.130018    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995"} err="failed to get container status \"e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995\": rpc error: code = NotFound desc = could not find container \"e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995\": container with ID starting with e15999247ce8b252a679343c092e0bf07a9cafbd95e48363e990a551d6612995 not found: ID does not exist"
Nov 01 23:02:24.130082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.130059    4837 scope.go:122] "RemoveContainer" containerID="cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc"
Nov 01 23:02:24.138950 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.138929    4837 scope.go:122] "RemoveContainer" containerID="cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc"
Nov 01 23:02:24.139288 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:02:24.139264    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc\": container with ID starting with cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc not found: ID does not exist" containerID="cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc"
Nov 01 23:02:24.139363 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.139297    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc"} err="failed to get container status \"cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc\": rpc error: code = NotFound desc = could not find container \"cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc\": container with ID starting with cbd5eba2b99d1acda8cebbb8d33b3476d039593bb67fa48993494d2c417414bc not found: ID does not exist"
Nov 01 23:02:24.155357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.155333    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.155357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.155357    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.155485 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.155368    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e14863a-1388-4ccd-8c62-8630cc3ba882-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.155485 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.155376    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.155485 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.155386    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e14863a-1388-4ccd-8c62-8630cc3ba882-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.155485 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.155394    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs9k2\" (UniqueName: \"kubernetes.io/projected/8e14863a-1388-4ccd-8c62-8630cc3ba882-kube-api-access-xs9k2\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.274225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.274188    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"]
Nov 01 23:02:24.283563 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.283527    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-db7p9"]
Nov 01 23:02:24.383531 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.383454    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"]
Nov 01 23:02:24.387269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.387240    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-s2dnh"]
Nov 01 23:02:24.533976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.533940    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" path="/var/lib/kubelet/pods/28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95/volumes"
Nov 01 23:02:24.534506 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:24.534485    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" path="/var/lib/kubelet/pods/8e14863a-1388-4ccd-8c62-8630cc3ba882/volumes"
Nov 01 23:02:27.144862 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:27.144829    4837 generic.go:356] "Generic (PLEG): container finished" podID="b5463180-2162-4d32-b896-2ddd39e1b170" containerID="6c9787cb3e6af8a184d49c392282fe7d8c10a539e2b1d61790467708e0c75d96" exitCode=0
Nov 01 23:02:27.145311 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:27.144905    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" event={"ID":"b5463180-2162-4d32-b896-2ddd39e1b170","Type":"ContainerDied","Data":"6c9787cb3e6af8a184d49c392282fe7d8c10a539e2b1d61790467708e0c75d96"}
Nov 01 23:02:27.145360 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:27.145340    4837 scope.go:122] "RemoveContainer" containerID="6c9787cb3e6af8a184d49c392282fe7d8c10a539e2b1d61790467708e0c75d96"
Nov 01 23:02:27.147075 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:27.147034    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:02:27.147153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:27.147134    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:02:27.147191 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:27.147178    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:02:28.173795 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:28.171304    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" event={"ID":"b5463180-2162-4d32-b896-2ddd39e1b170","Type":"ContainerStarted","Data":"ade887165161bcd91e2d20d1fe660aff4cbb7a7d556932171ad3ab396aaefebe"}
Nov 01 23:02:28.429258 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:28.429181    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:02:28.438670 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:28.438159    4837 patch_prober.go:29] interesting pod/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 01 23:02:28.438670 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:28.438237    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 01 23:02:29.428623 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:29.428589    4837 patch_prober.go:29] interesting pod/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 01 23:02:29.429067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:29.428648    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 01 23:02:30.210856 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:30.210825    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:02:30.428754 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:30.428723    4837 patch_prober.go:29] interesting pod/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 01 23:02:30.429210 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:30.428790    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 01 23:02:31.429360 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:31.429329    4837 patch_prober.go:29] interesting pod/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 01 23:02:31.429762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:31.429380    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 01 23:02:32.429122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:32.429082    4837 patch_prober.go:29] interesting pod/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 01 23:02:32.429335 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:32.429153    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 01 23:02:33.434139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:33.434107    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:02:33.435127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:33.435106    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:02:47.821475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:47.821443    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"]
Nov 01 23:02:47.822083 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:47.821651    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c" podUID="d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" containerName="maas-api" containerID="cri-o://57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17" gracePeriod=30
Nov 01 23:02:48.144753 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.144728    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 23:02:48.278617 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.278557    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-kube-api-access-dww4r\" (UniqueName: \"kubernetes.io/projected/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-kube-api-access-dww4r\") pod \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") "
Nov 01 23:02:48.278814 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.278629    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-openshift-service-ca\") pod \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") "
Nov 01 23:02:48.278814 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.278788    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-metrics-certs\") pod \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") "
Nov 01 23:02:48.278814 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.278829    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls\") pod \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\" (UID: \"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2\") "
Nov 01 23:02:48.281062 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.281010    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-openshift-service-ca" pod "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" (UID: "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:48.299614 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.299544    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls" pod "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" (UID: "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:48.309289 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.309221    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-metrics-certs" pod "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" (UID: "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:48.326470 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.326435    4837 generic.go:356] "Generic (PLEG): container finished" podID="d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" containerID="57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17" exitCode=0
Nov 01 23:02:48.326703 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.326685    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c" event={"ID":"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2","Type":"ContainerDied","Data":"57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17"}
Nov 01 23:02:48.326840 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.326826    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c" event={"ID":"d4f17758-fa93-4a5b-b5a7-ff38029a3ab2","Type":"ContainerDied","Data":"a8f19fc1b133f3d9f31e28b8c2d0e367bb629f90930825c42001dbbc8f214a29"}
Nov 01 23:02:48.326945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.326933    4837 scope.go:122] "RemoveContainer" containerID="57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17"
Nov 01 23:02:48.327225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.327206    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"
Nov 01 23:02:48.369128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.366819    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-kube-api-access-dww4r" pod "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" (UID: "d4f17758-fa93-4a5b-b5a7-ff38029a3ab2"). InnerVolumeSpecName "kube-api-access-dww4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:48.390582 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.390536    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dww4r\" (UniqueName: \"kubernetes.io/projected/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-kube-api-access-dww4r\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:48.390582 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.390576    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:48.390874 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.390593    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:48.390874 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.390607    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:48.391002 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.390962    4837 scope.go:122] "RemoveContainer" containerID="57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17"
Nov 01 23:02:48.391609 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:02:48.391575    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17\": container with ID starting with 57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17 not found: ID does not exist" containerID="57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17"
Nov 01 23:02:48.391785 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.391617    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17"} err="failed to get container status \"57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17\": rpc error: code = NotFound desc = could not find container \"57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17\": container with ID starting with 57b5b61682883606c664ce7b470235464f7424859791c59e2f425ddda5ea0c17 not found: ID does not exist"
Nov 01 23:02:48.735988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.735959    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"]
Nov 01 23:02:48.830061 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:48.829254    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-584561-f985548c7-gmd5c"]
Nov 01 23:02:48.974347 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:02:48.974313    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:02:50.528003 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:50.527690    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" path="/var/lib/kubelet/pods/d4f17758-fa93-4a5b-b5a7-ff38029a3ab2/volumes"
Nov 01 23:02:57.049273 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:57.049238    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"]
Nov 01 23:02:57.049668 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:57.049445    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy" containerID="cri-o://05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310" gracePeriod=30
Nov 01 23:02:59.777640 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:59.777601    4837 patch_prober.go:29] interesting pod/e2e-api-b-584561-openshift-default-95c6887c-vdglb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:02:59.778017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:02:59.777656    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:03:02.323540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.323487    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 23:03:02.410180 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410154    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istiod-ca-cert\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.410342 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410239    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-kube-api-access-l6jvk\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-kube-api-access-l6jvk\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.410342 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410278    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-token\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-token\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.410342 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410334    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-credential-socket\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.411074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410613    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-credential-socket" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:02.411074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410636    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-podinfo\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.411074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410774    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-certs\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.411074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410838    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-data\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.411074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410547    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istiod-ca-cert" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:02.411074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.410887    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-socket\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.411414 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.411209    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-envoy\") pod \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\" (UID: \"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4\") "
Nov 01 23:03:02.411477 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.411417    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-certs" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:02.411692 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.411667    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-data" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:02.411916 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.411885    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-socket" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:02.412078 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.411994    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.412078 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.412017    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.412078 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.412031    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.412276 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.412079    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.412994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.412967    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-podinfo" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:03:02.413209 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.413187    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-kube-api-access-l6jvk" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "kube-api-access-l6jvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:02.413291 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.413271    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-token" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:02.413993 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.413967    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-envoy" pod "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" (UID: "d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:02.419669 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.419642    4837 generic.go:356] "Generic (PLEG): container finished" podID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerID="05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310" exitCode=0
Nov 01 23:03:02.419756 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.419694    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"
Nov 01 23:03:02.419756 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.419719    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" event={"ID":"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4","Type":"ContainerDied","Data":"05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310"}
Nov 01 23:03:02.419841 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.419776    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb" event={"ID":"d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4","Type":"ContainerDied","Data":"6caef8b3cac5352f6abecf23d257a33dcdf2ebfd2e935ff63ec90f9e43f54425"}
Nov 01 23:03:02.419841 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.419803    4837 scope.go:122] "RemoveContainer" containerID="05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310"
Nov 01 23:03:02.442612 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.442592    4837 scope.go:122] "RemoveContainer" containerID="3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846"
Nov 01 23:03:02.453429 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.452752    4837 scope.go:122] "RemoveContainer" containerID="05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310"
Nov 01 23:03:02.455038 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:03:02.455010    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310\": container with ID starting with 05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310 not found: ID does not exist" containerID="05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310"
Nov 01 23:03:02.455146 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.455078    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310"} err="failed to get container status \"05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310\": rpc error: code = NotFound desc = could not find container \"05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310\": container with ID starting with 05e923aa924eb43ed478ffec6a5d6473cdc460793e8718b0e1a7e189c1056310 not found: ID does not exist"
Nov 01 23:03:02.455146 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.455100    4837 scope.go:122] "RemoveContainer" containerID="3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846"
Nov 01 23:03:02.455357 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:03:02.455342    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846\": container with ID starting with 3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846 not found: ID does not exist" containerID="3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846"
Nov 01 23:03:02.455402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.455361    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846"} err="failed to get container status \"3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846\": rpc error: code = NotFound desc = could not find container \"3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846\": container with ID starting with 3a1a9f3f69474af310880d62ae8c7479116fbf1c9c689c454232b049aac29846 not found: ID does not exist"
Nov 01 23:03:02.475764 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.475733    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"]
Nov 01 23:03:02.513862 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.513829    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6jvk\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-kube-api-access-l6jvk\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.513862 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.513863    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.514070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.513873    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.514070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.513882    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.514070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.513891    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:02.529892 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.529862    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"]
Nov 01 23:03:02.733506 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.733466    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"]
Nov 01 23:03:02.738884 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:02.738859    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-584561-openshift-default-95c6887c-vdglb"]
Nov 01 23:03:04.526501 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:04.526466    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" path="/var/lib/kubelet/pods/d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4/volumes"
Nov 01 23:03:05.039728 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.039698    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"]
Nov 01 23:03:05.040393 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040371    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy"
Nov 01 23:03:05.040393 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040390    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040401    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" containerName="maas-api"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040406    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" containerName="maas-api"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040435    4837 cpu_manager.go:414] "containerMap: removing container" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040443    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040457    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040463    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040482    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040486    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040499    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy"
Nov 01 23:03:05.040568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040504    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy"
Nov 01 23:03:05.040878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040588    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy"
Nov 01 23:03:05.040878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040596    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f17758-fa93-4a5b-b5a7-ff38029a3ab2" containerName="maas-api"
Nov 01 23:03:05.040878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040608    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ae3b0c-b48c-4fcc-83ef-0e18aad2fe95" containerName="istio-proxy"
Nov 01 23:03:05.040878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040615    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e389acf-e06f-4432-bf17-2d302059ef43" containerName="main"
Nov 01 23:03:05.040878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040623    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40070f1-fc8e-4b0e-916f-eb2bc2f0d3a4" containerName="istio-proxy"
Nov 01 23:03:05.040878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.040630    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e14863a-1388-4ccd-8c62-8630cc3ba882" containerName="istio-proxy"
Nov 01 23:03:05.044366 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.044344    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.093231 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.093207    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-d6101b-openshift-default-dockercfg-jbnc7\""
Nov 01 23:03:05.143703 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.143672    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"]
Nov 01 23:03:05.146584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.146555    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.146689 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.146587    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.146689 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.146631    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.146842 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.146816    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2661a9df-eef7-4201-bdb9-0f6311a11c88-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.146943 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.146925    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.147003 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.146989    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.147113 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.147092    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428dq\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-kube-api-access-428dq\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.147250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.147234    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.147289 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.147274    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.248921 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.248884    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.248956    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2661a9df-eef7-4201-bdb9-0f6311a11c88-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249023    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249259 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249124    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249259 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249163    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428dq\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-kube-api-access-428dq\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249280    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249319    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249466 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249373    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249466 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249399    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249571 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249537    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249625 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249590    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249862 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249838    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2661a9df-eef7-4201-bdb9-0f6311a11c88-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249842    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.249976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.249893    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.251576 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.251554    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.251662 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.251600    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.303007 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.302947    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.303215 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.303199    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-428dq\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-kube-api-access-428dq\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.357273 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.357248    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:05.782525 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.782496    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"]
Nov 01 23:03:05.783252 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:03:05.783229    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2661a9df_eef7_4201_bdb9_0f6311a11c88.slice/crio-dd91d6b137e453a75d990ba72dfd9db149ddbac5c0b153b09a7450d2f90b478a WatchSource:0}: Error finding container dd91d6b137e453a75d990ba72dfd9db149ddbac5c0b153b09a7450d2f90b478a: Status 404 returned error can't find the container with id dd91d6b137e453a75d990ba72dfd9db149ddbac5c0b153b09a7450d2f90b478a
Nov 01 23:03:05.785918 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.785881    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:05.786035 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.785966    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:05.786035 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:05.786010    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:06.458373 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:06.458338    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" event={"ID":"2661a9df-eef7-4201-bdb9-0f6311a11c88","Type":"ContainerStarted","Data":"613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e"}
Nov 01 23:03:06.458566 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:06.458392    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" event={"ID":"2661a9df-eef7-4201-bdb9-0f6311a11c88","Type":"ContainerStarted","Data":"dd91d6b137e453a75d990ba72dfd9db149ddbac5c0b153b09a7450d2f90b478a"}
Nov 01 23:03:06.570391 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:06.570339    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" podStartSLOduration=2.570322309 podStartE2EDuration="2.570322309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:06.568175467 +0000 UTC m=+2368.157374528" watchObservedRunningTime="2026-11-01 23:03:06.570322309 +0000 UTC m=+2368.159521375"
Nov 01 23:03:07.358151 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:07.358109    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:07.360167 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:07.360138    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 01 23:03:07.360304 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:07.360194    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 01 23:03:08.358546 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:08.358515    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 01 23:03:08.358963 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:08.358572    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 01 23:03:09.358157 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:09.358122    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 01 23:03:09.358355 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:09.358178    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 01 23:03:09.512978 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:09.512941    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"]
Nov 01 23:03:10.358074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:10.358027    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 01 23:03:10.358285 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:10.358098    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 01 23:03:11.361065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:11.360418    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 01 23:03:11.361065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:11.360483    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 01 23:03:12.358661 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:12.358621    4837 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 01 23:03:12.358824 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:12.358706    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 01 23:03:13.362879 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:13.362850    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:13.363379 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:13.363249    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:13.363986 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:13.363969    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:03:14.514238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:14.514204    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d" podUID="a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" containerName="payload-processing" containerID="cri-o://db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c" gracePeriod=25
Nov 01 23:03:17.496233 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.496194    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" containerID="cri-o://064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73" gracePeriod=45
Nov 01 23:03:17.519747 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.519720    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused"
Nov 01 23:03:17.552551 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.552518    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" containerID="cri-o://f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd" gracePeriod=45
Nov 01 23:03:17.691300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.691267    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused"
Nov 01 23:03:17.872019 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.871993    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 23:03:17.881149 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.881125    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.991725    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-home\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-home\") pod \"c1acf4ec-d463-4380-be2b-2da98bf7d984\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.991806    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-model-cache\") pod \"f066a460-1722-4886-9978-8f677610c83b\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.991849    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1acf4ec-d463-4380-be2b-2da98bf7d984-kube-api-access-ngln6\" (UniqueName: \"kubernetes.io/projected/c1acf4ec-d463-4380-be2b-2da98bf7d984-kube-api-access-ngln6\") pod \"c1acf4ec-d463-4380-be2b-2da98bf7d984\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.991890    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-tmp-dir\") pod \"f066a460-1722-4886-9978-8f677610c83b\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.991924    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-dshm\") pod \"f066a460-1722-4886-9978-8f677610c83b\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.991979    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-dshm\") pod \"c1acf4ec-d463-4380-be2b-2da98bf7d984\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.992007    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-home\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-home\") pod \"f066a460-1722-4886-9978-8f677610c83b\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.992036    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f066a460-1722-4886-9978-8f677610c83b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f066a460-1722-4886-9978-8f677610c83b-tls-certs\") pod \"f066a460-1722-4886-9978-8f677610c83b\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.992133    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-tmp-dir\") pod \"c1acf4ec-d463-4380-be2b-2da98bf7d984\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.992173    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f066a460-1722-4886-9978-8f677610c83b-kube-api-access-96w9t\" (UniqueName: \"kubernetes.io/projected/f066a460-1722-4886-9978-8f677610c83b-kube-api-access-96w9t\") pod \"f066a460-1722-4886-9978-8f677610c83b\" (UID: \"f066a460-1722-4886-9978-8f677610c83b\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.992203    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1acf4ec-d463-4380-be2b-2da98bf7d984-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1acf4ec-d463-4380-be2b-2da98bf7d984-tls-certs\") pod \"c1acf4ec-d463-4380-be2b-2da98bf7d984\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") "
Nov 01 23:03:17.993197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.992349    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-model-cache\") pod \"c1acf4ec-d463-4380-be2b-2da98bf7d984\" (UID: \"c1acf4ec-d463-4380-be2b-2da98bf7d984\") "
Nov 01 23:03:17.994095 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.994060    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-model-cache" pod "c1acf4ec-d463-4380-be2b-2da98bf7d984" (UID: "c1acf4ec-d463-4380-be2b-2da98bf7d984"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.994643 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.994616    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-model-cache" pod "f066a460-1722-4886-9978-8f677610c83b" (UID: "f066a460-1722-4886-9978-8f677610c83b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.994811 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.994765    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-tmp-dir" pod "f066a460-1722-4886-9978-8f677610c83b" (UID: "f066a460-1722-4886-9978-8f677610c83b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.994921 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.994769    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-tmp-dir" pod "c1acf4ec-d463-4380-be2b-2da98bf7d984" (UID: "c1acf4ec-d463-4380-be2b-2da98bf7d984"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.995015 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.994971    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-home" pod "f066a460-1722-4886-9978-8f677610c83b" (UID: "f066a460-1722-4886-9978-8f677610c83b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.995459 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.995417    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-home" pod "c1acf4ec-d463-4380-be2b-2da98bf7d984" (UID: "c1acf4ec-d463-4380-be2b-2da98bf7d984"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.995581 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.995495    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-dshm" pod "f066a460-1722-4886-9978-8f677610c83b" (UID: "f066a460-1722-4886-9978-8f677610c83b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.997483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.997445    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1acf4ec-d463-4380-be2b-2da98bf7d984-tls-certs" pod "c1acf4ec-d463-4380-be2b-2da98bf7d984" (UID: "c1acf4ec-d463-4380-be2b-2da98bf7d984"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:17.997844 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.997819    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f066a460-1722-4886-9978-8f677610c83b-tls-certs" pod "f066a460-1722-4886-9978-8f677610c83b" (UID: "f066a460-1722-4886-9978-8f677610c83b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:17.997965 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:17.997942    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f066a460-1722-4886-9978-8f677610c83b-kube-api-access-96w9t" pod "f066a460-1722-4886-9978-8f677610c83b" (UID: "f066a460-1722-4886-9978-8f677610c83b"). InnerVolumeSpecName "kube-api-access-96w9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:18.000142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.000068    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1acf4ec-d463-4380-be2b-2da98bf7d984-kube-api-access-ngln6" pod "c1acf4ec-d463-4380-be2b-2da98bf7d984" (UID: "c1acf4ec-d463-4380-be2b-2da98bf7d984"). InnerVolumeSpecName "kube-api-access-ngln6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:18.000395 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.000358    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-dshm" pod "c1acf4ec-d463-4380-be2b-2da98bf7d984" (UID: "c1acf4ec-d463-4380-be2b-2da98bf7d984"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:18.094922 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.094783    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.094922 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.094819    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.094922 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.094834    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095258    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngln6\" (UniqueName: \"kubernetes.io/projected/c1acf4ec-d463-4380-be2b-2da98bf7d984-kube-api-access-ngln6\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095290    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095306    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095319    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095332    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f066a460-1722-4886-9978-8f677610c83b-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095345    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f066a460-1722-4886-9978-8f677610c83b-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095358    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1acf4ec-d463-4380-be2b-2da98bf7d984-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095372    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96w9t\" (UniqueName: \"kubernetes.io/projected/f066a460-1722-4886-9978-8f677610c83b-kube-api-access-96w9t\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.095405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.095387    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1acf4ec-d463-4380-be2b-2da98bf7d984-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.557600 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.557568    4837 generic.go:356] "Generic (PLEG): container finished" podID="f066a460-1722-4886-9978-8f677610c83b" containerID="f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd" exitCode=0
Nov 01 23:03:18.558020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.557617    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"
Nov 01 23:03:18.558020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.557651    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" event={"ID":"f066a460-1722-4886-9978-8f677610c83b","Type":"ContainerDied","Data":"f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd"}
Nov 01 23:03:18.558020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.557689    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd" event={"ID":"f066a460-1722-4886-9978-8f677610c83b","Type":"ContainerDied","Data":"4108d4fd7fb66ecc19534d05f126648b1a36183d230d880ae159774170128bd0"}
Nov 01 23:03:18.558020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.557705    4837 scope.go:122] "RemoveContainer" containerID="f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd"
Nov 01 23:03:18.559141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.559110    4837 generic.go:356] "Generic (PLEG): container finished" podID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerID="064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73" exitCode=0
Nov 01 23:03:18.559141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.559137    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" event={"ID":"c1acf4ec-d463-4380-be2b-2da98bf7d984","Type":"ContainerDied","Data":"064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73"}
Nov 01 23:03:18.559298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.559159    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"
Nov 01 23:03:18.559298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.559177    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw" event={"ID":"c1acf4ec-d463-4380-be2b-2da98bf7d984","Type":"ContainerDied","Data":"4e47b7cfd1c96b0ce327083e3757e2f483c0bd76bde0c42549ec8198c40e7c1b"}
Nov 01 23:03:18.570733 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.570707    4837 scope.go:122] "RemoveContainer" containerID="f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd"
Nov 01 23:03:18.571004 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:03:18.570981    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd\": container with ID starting with f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd not found: ID does not exist" containerID="f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd"
Nov 01 23:03:18.571110 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.571012    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd"} err="failed to get container status \"f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd\": rpc error: code = NotFound desc = could not find container \"f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd\": container with ID starting with f7f4853871bf737bf559e802e82dd4b86e23e4f2bcd8525abeac30af1d68acdd not found: ID does not exist"
Nov 01 23:03:18.571110 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.571029    4837 scope.go:122] "RemoveContainer" containerID="064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73"
Nov 01 23:03:18.579542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.579527    4837 scope.go:122] "RemoveContainer" containerID="064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73"
Nov 01 23:03:18.579832 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:03:18.579813    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73\": container with ID starting with 064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73 not found: ID does not exist" containerID="064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73"
Nov 01 23:03:18.579884 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.579837    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73"} err="failed to get container status \"064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73\": rpc error: code = NotFound desc = could not find container \"064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73\": container with ID starting with 064b0ed3705917b532b11b27ecbf2979b4b0b28079d4a9d8f8e11fcc0b64ad73 not found: ID does not exist"
Nov 01 23:03:18.714312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.714277    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"]
Nov 01 23:03:18.720614 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.720591    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/simulator-w3-e71d55-kserve-54d4c876bc-4qcmd"]
Nov 01 23:03:18.815124 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:03:18.815021    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:03:18.822725 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.822659    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"]
Nov 01 23:03:18.826174 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:18.826151    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-w3-e71d55-kserve-5976ffff69-rm5rw"]
Nov 01 23:03:20.525142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:20.525114    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" path="/var/lib/kubelet/pods/c1acf4ec-d463-4380-be2b-2da98bf7d984/volumes"
Nov 01 23:03:20.525554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:20.525539    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f066a460-1722-4886-9978-8f677610c83b" path="/var/lib/kubelet/pods/f066a460-1722-4886-9978-8f677610c83b/volumes"
Nov 01 23:03:22.024850 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.024809    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"]
Nov 01 23:03:22.025565 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.025539    4837 cpu_manager.go:414] "containerMap: removing container" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main"
Nov 01 23:03:22.025565 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.025561    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main"
Nov 01 23:03:22.025741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.025587    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main"
Nov 01 23:03:22.025741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.025595    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main"
Nov 01 23:03:22.025741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.025703    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1acf4ec-d463-4380-be2b-2da98bf7d984" containerName="main"
Nov 01 23:03:22.025741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.025715    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f066a460-1722-4886-9978-8f677610c83b" containerName="main"
Nov 01 23:03:22.030143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.030123    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.081424 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.081390    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4af44d-openshift-default-dockercfg-k4qhb\""
Nov 01 23:03:22.130004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.129974    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"]
Nov 01 23:03:22.146470 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.146442    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-podinfo\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.146647 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.146484    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-certs\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.146647 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.146508    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-credential-socket\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.146774 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.146697    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-token\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.146774 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.146748    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-socket\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.146935 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.146912    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165cb48-bf04-4815-a668-31a3af4f64ed-istiod-ca-cert\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.146998 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.146981    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcbc8\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-kube-api-access-qcbc8\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.147056 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.147017    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-envoy\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.147161 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.147145    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-data\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248468    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-podinfo\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248698 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248536    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-certs\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248698 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248568    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-credential-socket\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248698 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248649    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-token\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248698 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248689    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-socket\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248955 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248806    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165cb48-bf04-4815-a668-31a3af4f64ed-istiod-ca-cert\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248955 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248867    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcbc8\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-kube-api-access-qcbc8\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248955 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248894    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-envoy\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.248955 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248941    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-data\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.249217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.248986    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-certs\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.249217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.249059    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-credential-socket\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.249397 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.249375    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-data\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.249452 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.249412    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-socket\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.250018 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.249993    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165cb48-bf04-4815-a668-31a3af4f64ed-istiod-ca-cert\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.251287 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.251262    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-envoy\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.251287 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.251279    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-podinfo\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.306399 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.306321    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcbc8\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-kube-api-access-qcbc8\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.332960 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.332876    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-token\") pod \"e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") " pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.343455 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.343422    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:22.765021 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.764990    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"]
Nov 01 23:03:22.798461 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:03:22.798433    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0165cb48_bf04_4815_a668_31a3af4f64ed.slice/crio-beadd174fa17c3e9efdd8672ba9690e658a941d08e8943c060b9700a31131bfb WatchSource:0}: Error finding container beadd174fa17c3e9efdd8672ba9690e658a941d08e8943c060b9700a31131bfb: Status 404 returned error can't find the container with id beadd174fa17c3e9efdd8672ba9690e658a941d08e8943c060b9700a31131bfb
Nov 01 23:03:22.801528 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.801486    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:22.801620 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.801575    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:22.801668 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:22.801617    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:23.594415 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:23.594379    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" event={"ID":"0165cb48-bf04-4815-a668-31a3af4f64ed","Type":"ContainerStarted","Data":"c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6"}
Nov 01 23:03:23.594811 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:23.594446    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" event={"ID":"0165cb48-bf04-4815-a668-31a3af4f64ed","Type":"ContainerStarted","Data":"beadd174fa17c3e9efdd8672ba9690e658a941d08e8943c060b9700a31131bfb"}
Nov 01 23:03:23.706210 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:23.706163    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" podStartSLOduration=2.706150169 podStartE2EDuration="2.706150169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:23.704476361 +0000 UTC m=+2385.293675421" watchObservedRunningTime="2026-11-01 23:03:23.706150169 +0000 UTC m=+2385.295349229"
Nov 01 23:03:24.344744 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:24.344711    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:24.350754 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:24.350732    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:24.463140 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:24.463105    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d" podUID="a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:03:24.608445 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:24.608357    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:24.609449 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:24.609430    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:03:28.166899 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.166790    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:03:28.175646 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.175617    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"]
Nov 01 23:03:28.180575 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.180530    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.257731 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.257443    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cf46c7-openshift-default-dockercfg-7lwd9\""
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259113    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-certs\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259169    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-envoy\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259243    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-podinfo\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259309    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsg8\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-kube-api-access-5vsg8\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259385    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istiod-ca-cert\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259420    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-credential-socket\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259575    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-data\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259614    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-socket\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.264387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.259643    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-token\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.400346 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.384618    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-credential-socket\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.400796 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.400751    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-data\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.401006 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.400989    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-socket\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.401144 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.401130    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-token\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.401291 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.401274    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-certs\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.401430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.401415    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-envoy\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.401616 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.401600    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-podinfo\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.419115 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.401778    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsg8\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-kube-api-access-5vsg8\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.419480 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.419449    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istiod-ca-cert\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.419758 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.403624    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-data\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.419895 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.403861    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-socket\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.462353 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.421365    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-certs\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.484391 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.403349    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-credential-socket\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.484630 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.439442    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istiod-ca-cert\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.484745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.439895    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-podinfo\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.484860 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.484249    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"]
Nov 01 23:03:28.580402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.580377    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-envoy\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.580750 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.580723    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-token\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.584412 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.584383    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsg8\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-kube-api-access-5vsg8\") pod \"e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") " pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:28.616266 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:28.616237    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:29.299240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:29.299209    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"]
Nov 01 23:03:29.300097 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:03:29.300070    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ebe69f_f0df_4388_b1d1_b15c1e43351e.slice/crio-4b665cab346b65c91a04b41d390aa2bac900d6dec696703300c0ba471d2086f1 WatchSource:0}: Error finding container 4b665cab346b65c91a04b41d390aa2bac900d6dec696703300c0ba471d2086f1: Status 404 returned error can't find the container with id 4b665cab346b65c91a04b41d390aa2bac900d6dec696703300c0ba471d2086f1
Nov 01 23:03:29.302883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:29.302849    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:29.302974 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:29.302913    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:29.302974 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:29.302941    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:29.768460 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:29.768423    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" event={"ID":"83ebe69f-f0df-4388-b1d1-b15c1e43351e","Type":"ContainerStarted","Data":"fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b"}
Nov 01 23:03:29.768639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:29.768488    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" event={"ID":"83ebe69f-f0df-4388-b1d1-b15c1e43351e","Type":"ContainerStarted","Data":"4b665cab346b65c91a04b41d390aa2bac900d6dec696703300c0ba471d2086f1"}
Nov 01 23:03:29.935063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:29.934316    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" podStartSLOduration=1.9342964440000001 podStartE2EDuration="1.934296444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:29.933515361 +0000 UTC m=+2391.522714426" watchObservedRunningTime="2026-11-01 23:03:29.934296444 +0000 UTC m=+2391.523495506"
Nov 01 23:03:30.617070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:30.617030    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:30.622615 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:30.622595    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:30.776647 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:30.776621    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:30.777470 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:30.777451    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:03:34.464273 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.464231    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d" podUID="a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:03:34.787027 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.787002    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 23:03:34.794637 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.794613    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-kube-api-access-z7vnk\") pod \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\" (UID: \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\") "
Nov 01 23:03:34.794768 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.794690    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-plugins-config-volume\") pod \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\" (UID: \"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7\") "
Nov 01 23:03:34.795015 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.794991    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-plugins-config-volume" pod "a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" (UID: "a7ac5f8f-d582-4e2a-b94b-2f727121a5f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:34.795472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.795450    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:34.796942 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.796917    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-kube-api-access-z7vnk" pod "a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" (UID: "a7ac5f8f-d582-4e2a-b94b-2f727121a5f7"). InnerVolumeSpecName "kube-api-access-z7vnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:34.812325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.812292    4837 generic.go:356] "Generic (PLEG): container finished" podID="a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" containerID="db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c" exitCode=0
Nov 01 23:03:34.812433 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.812330    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"
Nov 01 23:03:34.812433 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.812373    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d" event={"ID":"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7","Type":"ContainerDied","Data":"db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c"}
Nov 01 23:03:34.812433 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.812426    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d" event={"ID":"a7ac5f8f-d582-4e2a-b94b-2f727121a5f7","Type":"ContainerDied","Data":"aec140513228dc823c2ec784168f87797b7d21e7ad6befd0d2540d63cb39a293"}
Nov 01 23:03:34.812587 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.812449    4837 scope.go:122] "RemoveContainer" containerID="db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c"
Nov 01 23:03:34.822958 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.822936    4837 scope.go:122] "RemoveContainer" containerID="db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c"
Nov 01 23:03:34.823251 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:03:34.823229    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c\": container with ID starting with db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c not found: ID does not exist" containerID="db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c"
Nov 01 23:03:34.823317 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.823259    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c"} err="failed to get container status \"db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c\": rpc error: code = NotFound desc = could not find container \"db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c\": container with ID starting with db9add8931bcf62673458e51f831557e1a2139bca50b888af87f1fdd5c33506c not found: ID does not exist"
Nov 01 23:03:34.897208 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:34.897179    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7vnk\" (UniqueName: \"kubernetes.io/projected/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7-kube-api-access-z7vnk\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:35.023456 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:35.023370    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"]
Nov 01 23:03:35.028929 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:35.028901    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-e71d55-66fbf64559-wg69d"]
Nov 01 23:03:35.822213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:35.822139    4837 generic.go:356] "Generic (PLEG): container finished" podID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerID="480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df" exitCode=0
Nov 01 23:03:35.822641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:35.822207    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" event={"ID":"2b2f5e83-f48d-422e-a12c-65942e5ceb5f","Type":"ContainerDied","Data":"480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df"}
Nov 01 23:03:35.822641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:35.822629    4837 scope.go:122] "RemoveContainer" containerID="480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df"
Nov 01 23:03:35.824271 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:35.824244    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:35.824381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:35.824310    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:35.824381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:35.824360    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:03:36.525796 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:36.525764    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" path="/var/lib/kubelet/pods/a7ac5f8f-d582-4e2a-b94b-2f727121a5f7/volumes"
Nov 01 23:03:36.846551 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:36.846463    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" event={"ID":"2b2f5e83-f48d-422e-a12c-65942e5ceb5f","Type":"ContainerStarted","Data":"d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f"}
Nov 01 23:03:37.357141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:37.357104    4837 patch_prober.go:29] interesting pod/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 23:03:37.363986 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:37.363939    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 23:03:37.373936 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:37.357381    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:03:38.326972 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:38.326939    4837 patch_prober.go:29] interesting pod/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 23:03:38.327409 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:38.326989    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 23:03:38.890273 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:38.890244    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:03:38.891798 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:38.891770    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:03:39.327060 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:39.327016    4837 patch_prober.go:29] interesting pod/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 23:03:39.327459 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:39.327089    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 23:03:40.326805 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:40.326769    4837 patch_prober.go:29] interesting pod/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 23:03:40.326974 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:40.326828    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 23:03:41.326635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:41.326605    4837 patch_prober.go:29] interesting pod/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 23:03:41.327037 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:41.326653    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 23:03:41.866264 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:41.866234    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:03:42.330635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:42.330608    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:03:42.331609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:03:42.331588    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:03:48.813086 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:03:48.813034    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:04:18.819385 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:04:18.819347    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:04:24.705125 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:24.705095    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"]
Nov 01 23:04:24.705701 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:24.705364    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy" containerID="cri-o://c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7" gracePeriod=30
Nov 01 23:04:27.677425 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:27.677389    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"]
Nov 01 23:04:27.677924 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:27.677613    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy" containerID="cri-o://0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09" gracePeriod=30
Nov 01 23:04:29.484426 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:29.484393    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"]
Nov 01 23:04:29.484842 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:29.484620    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy" containerID="cri-o://613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e" gracePeriod=30
Nov 01 23:04:29.986414 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:29.986359    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 23:04:30.104662 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.104627    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-data\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.104875 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.104697    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istiod-ca-cert\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.104875 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.104734    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-kube-api-access-4s58t\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-kube-api-access-4s58t\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.104875 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.104785    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-certs\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.104875 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.104840    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-socket\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.105128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.104886    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-token\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-token\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.105128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.104947    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-credential-socket\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.105128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.105018    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-podinfo\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.105128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.105113    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-envoy\") pod \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\" (UID: \"6b56f6a2-6dad-405e-9d88-c30f0f9cee77\") "
Nov 01 23:04:30.105288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.105220    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istiod-ca-cert" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:30.105787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.105763    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.107629 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.107522    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-credential-socket" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:30.107629 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.107565    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-socket" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:30.107629 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.107606    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-certs" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:30.107840 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.107469    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-data" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:30.108469 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.108427    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-envoy" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:30.108612 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.108467    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-kube-api-access-4s58t" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "kube-api-access-4s58t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:30.108612 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.108500    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-token" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:30.109181 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.109146    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-podinfo" pod "6b56f6a2-6dad-405e-9d88-c30f0f9cee77" (UID: "6b56f6a2-6dad-405e-9d88-c30f0f9cee77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:04:30.154240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.154209    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"]
Nov 01 23:04:30.157300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.157276    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"]
Nov 01 23:04:30.207186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.207153    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.207186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.207185    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.207376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.207202    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.207376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.207216    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s58t\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-kube-api-access-4s58t\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.207376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.207231    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.207376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.207245    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.207376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.207258    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.207376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.207270    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b56f6a2-6dad-405e-9d88-c30f0f9cee77-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:30.266247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.266159    4837 generic.go:356] "Generic (PLEG): container finished" podID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerID="c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7" exitCode=0
Nov 01 23:04:30.266247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.266205    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" event={"ID":"6b56f6a2-6dad-405e-9d88-c30f0f9cee77","Type":"ContainerDied","Data":"c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7"}
Nov 01 23:04:30.266247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.266210    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"
Nov 01 23:04:30.266551 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.266250    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9" event={"ID":"6b56f6a2-6dad-405e-9d88-c30f0f9cee77","Type":"ContainerDied","Data":"cf5ad827866e035d4b5f02ed5fe80d5ad65340c7180b4812de5fa96bac78a2fa"}
Nov 01 23:04:30.266551 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.266274    4837 scope.go:122] "RemoveContainer" containerID="c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7"
Nov 01 23:04:30.278242 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.278222    4837 scope.go:122] "RemoveContainer" containerID="56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c"
Nov 01 23:04:30.288109 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.288093    4837 scope.go:122] "RemoveContainer" containerID="c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7"
Nov 01 23:04:30.288371 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:04:30.288349    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7\": container with ID starting with c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7 not found: ID does not exist" containerID="c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7"
Nov 01 23:04:30.288459 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.288383    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7"} err="failed to get container status \"c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7\": rpc error: code = NotFound desc = could not find container \"c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7\": container with ID starting with c8e575883b8ba6b91d99d394512411c446a3cf48bf9b31db3a8bb9f244be9ac7 not found: ID does not exist"
Nov 01 23:04:30.288459 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.288410    4837 scope.go:122] "RemoveContainer" containerID="56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c"
Nov 01 23:04:30.288675 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:04:30.288652    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c\": container with ID starting with 56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c not found: ID does not exist" containerID="56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c"
Nov 01 23:04:30.288732 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.288685    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c"} err="failed to get container status \"56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c\": rpc error: code = NotFound desc = could not find container \"56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c\": container with ID starting with 56bed32fee7cc7520a6c80a9740c553ef372e127f98cfc44e49027f9f78f9f3c not found: ID does not exist"
Nov 01 23:04:30.530605 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.530536    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"]
Nov 01 23:04:30.536472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:30.536450    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2b0866-openshift-default-5db989bbf-dbth9"]
Nov 01 23:04:32.527203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:32.527170    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" path="/var/lib/kubelet/pods/6b56f6a2-6dad-405e-9d88-c30f0f9cee77/volumes"
Nov 01 23:04:32.951243 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:32.951219    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 23:04:33.051344 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051274    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cddca7c6-47ca-440d-847e-1f1178de7051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cddca7c6-47ca-440d-847e-1f1178de7051-istio-podinfo\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.051344 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051324    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-kube-api-access-hx8p4\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-kube-api-access-hx8p4\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.051562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051407    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cddca7c6-47ca-440d-847e-1f1178de7051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cddca7c6-47ca-440d-847e-1f1178de7051-istiod-ca-cert\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.051562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051459    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-data\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.051562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051490    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-envoy\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.051562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051536    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-socket\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.051789 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051653    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-credential-socket\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.051789 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051724    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-certs\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.051789 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.051762    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-istio-token\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-istio-token\") pod \"cddca7c6-47ca-440d-847e-1f1178de7051\" (UID: \"cddca7c6-47ca-440d-847e-1f1178de7051\") "
Nov 01 23:04:33.052507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052013    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-data" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:33.052507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052089    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddca7c6-47ca-440d-847e-1f1178de7051-istiod-ca-cert" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:33.052507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052245    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-socket" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:33.052507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052417    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-credential-socket" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:33.052507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052482    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-certs" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:33.052870 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052655    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cddca7c6-47ca-440d-847e-1f1178de7051-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.052870 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052678    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.052870 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052694    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.052870 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052707    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.052870 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.052722    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.053882 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.053854    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-kube-api-access-hx8p4" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "kube-api-access-hx8p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:33.054700 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.054678    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cddca7c6-47ca-440d-847e-1f1178de7051-istio-podinfo" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:04:33.055325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.055300    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-envoy" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:33.063222 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.063190    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-istio-token" pod "cddca7c6-47ca-440d-847e-1f1178de7051" (UID: "cddca7c6-47ca-440d-847e-1f1178de7051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:33.154718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.154681    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.154718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.154719    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cddca7c6-47ca-440d-847e-1f1178de7051-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.154999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.154736    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx8p4\" (UniqueName: \"kubernetes.io/projected/cddca7c6-47ca-440d-847e-1f1178de7051-kube-api-access-hx8p4\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.154999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.154751    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cddca7c6-47ca-440d-847e-1f1178de7051-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.309653 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.309572    4837 generic.go:356] "Generic (PLEG): container finished" podID="cddca7c6-47ca-440d-847e-1f1178de7051" containerID="0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09" exitCode=0
Nov 01 23:04:33.309653 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.309643    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"
Nov 01 23:04:33.309837 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.309652    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" event={"ID":"cddca7c6-47ca-440d-847e-1f1178de7051","Type":"ContainerDied","Data":"0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09"}
Nov 01 23:04:33.309837 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.309744    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8" event={"ID":"cddca7c6-47ca-440d-847e-1f1178de7051","Type":"ContainerDied","Data":"a1ad3b50ae0a3fdf88c379a60ad9cc8d52f7f6f0110e7f3d4142fb7c36801725"}
Nov 01 23:04:33.309837 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.309774    4837 scope.go:122] "RemoveContainer" containerID="0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09"
Nov 01 23:04:33.321945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.321926    4837 scope.go:122] "RemoveContainer" containerID="65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9"
Nov 01 23:04:33.333828 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.333789    4837 scope.go:122] "RemoveContainer" containerID="0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09"
Nov 01 23:04:33.334341 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:04:33.334314    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09\": container with ID starting with 0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09 not found: ID does not exist" containerID="0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09"
Nov 01 23:04:33.334418 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.334349    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09"} err="failed to get container status \"0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09\": rpc error: code = NotFound desc = could not find container \"0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09\": container with ID starting with 0bd12b032501120a3624eb866fa82e134a852b2260aa89468495473110e66c09 not found: ID does not exist"
Nov 01 23:04:33.334418 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.334366    4837 scope.go:122] "RemoveContainer" containerID="65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9"
Nov 01 23:04:33.334636 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:04:33.334612    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9\": container with ID starting with 65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9 not found: ID does not exist" containerID="65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9"
Nov 01 23:04:33.334690 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.334633    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9"} err="failed to get container status \"65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9\": rpc error: code = NotFound desc = could not find container \"65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9\": container with ID starting with 65a328891c1a1bd7cb44dec5fcd452be55961b2ec7f6ffcee28887e5af4c5cc9 not found: ID does not exist"
Nov 01 23:04:33.466305 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.466269    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"]
Nov 01 23:04:33.469115 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:33.469089    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-njlw8"]
Nov 01 23:04:34.527669 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.527641    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" path="/var/lib/kubelet/pods/cddca7c6-47ca-440d-847e-1f1178de7051/volumes"
Nov 01 23:04:34.781057 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.781020    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:04:34.878699 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.878668    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-token\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-token\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.878838 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.878712    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-kube-api-access-428dq\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-kube-api-access-428dq\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.879283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.878856    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-data\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.879283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.878986    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-certs\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.879283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879099    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-credential-socket\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.879283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879141    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2661a9df-eef7-4201-bdb9-0f6311a11c88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2661a9df-eef7-4201-bdb9-0f6311a11c88-istiod-ca-cert\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.879283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879156    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-data" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.879283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879190    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-envoy\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.879283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879236    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-socket\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.879283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879280    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-podinfo\") pod \"2661a9df-eef7-4201-bdb9-0f6311a11c88\" (UID: \"2661a9df-eef7-4201-bdb9-0f6311a11c88\") "
Nov 01 23:04:34.879754 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879304    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-certs" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.879754 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879373    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-credential-socket" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.879754 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879512    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-socket" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.879754 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879736    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2661a9df-eef7-4201-bdb9-0f6311a11c88-istiod-ca-cert" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:34.879988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879854    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.879988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879875    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.879988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879890    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.879988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.879903    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.881078 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.881024    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-token" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:34.881178 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.881153    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-kube-api-access-428dq" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "kube-api-access-428dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:34.881519 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.881501    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-podinfo" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:04:34.881762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.881734    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-envoy" pod "2661a9df-eef7-4201-bdb9-0f6311a11c88" (UID: "2661a9df-eef7-4201-bdb9-0f6311a11c88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.981452 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.981415    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2661a9df-eef7-4201-bdb9-0f6311a11c88-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.981452 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.981449    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.981651 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.981464    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.981651 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.981478    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.981651 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:34.981492    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-428dq\" (UniqueName: \"kubernetes.io/projected/2661a9df-eef7-4201-bdb9-0f6311a11c88-kube-api-access-428dq\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:35.325122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.325085    4837 generic.go:356] "Generic (PLEG): container finished" podID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerID="613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e" exitCode=0
Nov 01 23:04:35.325324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.325141    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"
Nov 01 23:04:35.325324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.325166    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" event={"ID":"2661a9df-eef7-4201-bdb9-0f6311a11c88","Type":"ContainerDied","Data":"613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e"}
Nov 01 23:04:35.325324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.325222    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l" event={"ID":"2661a9df-eef7-4201-bdb9-0f6311a11c88","Type":"ContainerDied","Data":"dd91d6b137e453a75d990ba72dfd9db149ddbac5c0b153b09a7450d2f90b478a"}
Nov 01 23:04:35.325324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.325244    4837 scope.go:122] "RemoveContainer" containerID="613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e"
Nov 01 23:04:35.335755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.335740    4837 scope.go:122] "RemoveContainer" containerID="613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e"
Nov 01 23:04:35.336013 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:04:35.335996    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e\": container with ID starting with 613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e not found: ID does not exist" containerID="613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e"
Nov 01 23:04:35.336081 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.336021    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e"} err="failed to get container status \"613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e\": rpc error: code = NotFound desc = could not find container \"613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e\": container with ID starting with 613a4791ee63cb03cd57aa27b56d1be708d3e4309d99f40872f355625241646e not found: ID does not exist"
Nov 01 23:04:35.482743 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.482707    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"]
Nov 01 23:04:35.528717 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.528691    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-hzj7l"]
Nov 01 23:04:35.749420 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:35.749381    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:04:36.526291 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:36.526262    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" path="/var/lib/kubelet/pods/2661a9df-eef7-4201-bdb9-0f6311a11c88/volumes"
Nov 01 23:04:39.495756 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:39.495723    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"]
Nov 01 23:04:39.606863 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:39.606817    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"]
Nov 01 23:04:44.496851 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:44.496783    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw" podUID="3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" containerName="payload-processing" containerID="cri-o://9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147" gracePeriod=25
Nov 01 23:04:44.608001 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:44.607968    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw" podUID="76cef0b8-cc9d-4821-9456-e3189cf59ec5" containerName="payload-pre-processing" containerID="cri-o://c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9" gracePeriod=25
Nov 01 23:04:45.177468 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.177425    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" containerID="cri-o://54c0579e972a27c1a16250a3aa725248edee580ee03645915b0790266c7dd4af" gracePeriod=45
Nov 01 23:04:45.177643 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.177419    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main" containerID="cri-o://7260ba2a2f933ac509113e0b3dbd4d2e0855e28f806afb20c096edc3b62850b6" gracePeriod=45
Nov 01 23:04:45.419093 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.399062    4837 generic.go:356] "Generic (PLEG): container finished" podID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerID="54c0579e972a27c1a16250a3aa725248edee580ee03645915b0790266c7dd4af" exitCode=0
Nov 01 23:04:45.419093 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.399397    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" event={"ID":"cb48bd1c-2d40-4e8e-93ae-a6735406208d","Type":"ContainerDied","Data":"54c0579e972a27c1a16250a3aa725248edee580ee03645915b0790266c7dd4af"}
Nov 01 23:04:45.446735 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.446699    4837 generic.go:356] "Generic (PLEG): container finished" podID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerID="7260ba2a2f933ac509113e0b3dbd4d2e0855e28f806afb20c096edc3b62850b6" exitCode=0
Nov 01 23:04:45.446863 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.446837    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" event={"ID":"05a78acc-77a3-455d-9dd7-2e4f827fa20d","Type":"ContainerDied","Data":"7260ba2a2f933ac509113e0b3dbd4d2e0855e28f806afb20c096edc3b62850b6"}
Nov 01 23:04:45.570837 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.570510    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 23:04:45.603427 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.603398    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 23:04:45.653852 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.653825    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb48bd1c-2d40-4e8e-93ae-a6735406208d-kube-api-access-5jgz6\" (UniqueName: \"kubernetes.io/projected/cb48bd1c-2d40-4e8e-93ae-a6735406208d-kube-api-access-5jgz6\") pod \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") "
Nov 01 23:04:45.654020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.653870    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tmp-dir\") pod \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") "
Nov 01 23:04:45.654020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.653927    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-dshm\") pod \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") "
Nov 01 23:04:45.654020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.653962    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tls-certs\" (UniqueName: \"kubernetes.io/secret/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tls-certs\") pod \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") "
Nov 01 23:04:45.654225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654033    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-home\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-home\") pod \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") "
Nov 01 23:04:45.654225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654133    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tmp-dir\") pod \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") "
Nov 01 23:04:45.654225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654187    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tmp-dir" pod "cb48bd1c-2d40-4e8e-93ae-a6735406208d" (UID: "cb48bd1c-2d40-4e8e-93ae-a6735406208d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.654225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654211    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-home\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-home\") pod \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") "
Nov 01 23:04:45.654457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654256    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-model-cache\") pod \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") "
Nov 01 23:04:45.654457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654288    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-model-cache\") pod \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") "
Nov 01 23:04:45.654457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654324    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a78acc-77a3-455d-9dd7-2e4f827fa20d-kube-api-access-hwflm\" (UniqueName: \"kubernetes.io/projected/05a78acc-77a3-455d-9dd7-2e4f827fa20d-kube-api-access-hwflm\") pod \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") "
Nov 01 23:04:45.654457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654401    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-dshm\") pod \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\" (UID: \"05a78acc-77a3-455d-9dd7-2e4f827fa20d\") "
Nov 01 23:04:45.654457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654405    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-home" pod "05a78acc-77a3-455d-9dd7-2e4f827fa20d" (UID: "05a78acc-77a3-455d-9dd7-2e4f827fa20d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.654457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654454    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tls-certs\") pod \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\" (UID: \"cb48bd1c-2d40-4e8e-93ae-a6735406208d\") "
Nov 01 23:04:45.654784 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654479    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-home" pod "cb48bd1c-2d40-4e8e-93ae-a6735406208d" (UID: "cb48bd1c-2d40-4e8e-93ae-a6735406208d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.654784 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654697    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tmp-dir" pod "05a78acc-77a3-455d-9dd7-2e4f827fa20d" (UID: "05a78acc-77a3-455d-9dd7-2e4f827fa20d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.654784 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.654698    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-model-cache" pod "cb48bd1c-2d40-4e8e-93ae-a6735406208d" (UID: "cb48bd1c-2d40-4e8e-93ae-a6735406208d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.655233 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.655214    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.655331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.655236    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.655331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.655250    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.655331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.655263    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.655331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.655275    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.656557 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.656128    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-model-cache" pod "05a78acc-77a3-455d-9dd7-2e4f827fa20d" (UID: "05a78acc-77a3-455d-9dd7-2e4f827fa20d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.657357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.657180    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a78acc-77a3-455d-9dd7-2e4f827fa20d-kube-api-access-hwflm" pod "05a78acc-77a3-455d-9dd7-2e4f827fa20d" (UID: "05a78acc-77a3-455d-9dd7-2e4f827fa20d"). InnerVolumeSpecName "kube-api-access-hwflm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:45.657357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.657201    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-dshm" pod "05a78acc-77a3-455d-9dd7-2e4f827fa20d" (UID: "05a78acc-77a3-455d-9dd7-2e4f827fa20d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.657357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.657266    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-dshm" pod "cb48bd1c-2d40-4e8e-93ae-a6735406208d" (UID: "cb48bd1c-2d40-4e8e-93ae-a6735406208d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.657357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.657291    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tls-certs" pod "cb48bd1c-2d40-4e8e-93ae-a6735406208d" (UID: "cb48bd1c-2d40-4e8e-93ae-a6735406208d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:45.657647 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.657442    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb48bd1c-2d40-4e8e-93ae-a6735406208d-kube-api-access-5jgz6" pod "cb48bd1c-2d40-4e8e-93ae-a6735406208d" (UID: "cb48bd1c-2d40-4e8e-93ae-a6735406208d"). InnerVolumeSpecName "kube-api-access-5jgz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:45.658575 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.658536    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tls-certs" pod "05a78acc-77a3-455d-9dd7-2e4f827fa20d" (UID: "05a78acc-77a3-455d-9dd7-2e4f827fa20d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:45.756736 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.756654    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.756736 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.756693    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwflm\" (UniqueName: \"kubernetes.io/projected/05a78acc-77a3-455d-9dd7-2e4f827fa20d-kube-api-access-hwflm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.756736 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.756710    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a78acc-77a3-455d-9dd7-2e4f827fa20d-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.756736 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.756725    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb48bd1c-2d40-4e8e-93ae-a6735406208d-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.756736 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.756738    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgz6\" (UniqueName: \"kubernetes.io/projected/cb48bd1c-2d40-4e8e-93ae-a6735406208d-kube-api-access-5jgz6\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.757034 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.756749    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb48bd1c-2d40-4e8e-93ae-a6735406208d-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.757034 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:45.756756    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a78acc-77a3-455d-9dd7-2e4f827fa20d-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:46.453946 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.453871    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"
Nov 01 23:04:46.454133 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.453875    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" event={"ID":"cb48bd1c-2d40-4e8e-93ae-a6735406208d","Type":"ContainerDied","Data":"797c10e2c0940dc52928ec5aa5201f17687a1becbcfeee9edd5816683c43ec53"}
Nov 01 23:04:46.454133 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.454011    4837 scope.go:122] "RemoveContainer" containerID="54c0579e972a27c1a16250a3aa725248edee580ee03645915b0790266c7dd4af"
Nov 01 23:04:46.455504 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.455483    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"
Nov 01 23:04:46.455628 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.455488    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh" event={"ID":"05a78acc-77a3-455d-9dd7-2e4f827fa20d","Type":"ContainerDied","Data":"53ca418684bb9fc5a87df7e3b781b029bf215f7ec44ce415d2b3882d985e230e"}
Nov 01 23:04:46.463803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.463789    4837 scope.go:122] "RemoveContainer" containerID="7260ba2a2f933ac509113e0b3dbd4d2e0855e28f806afb20c096edc3b62850b6"
Nov 01 23:04:46.477854 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.477825    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw" podUID="3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:04:46.477963 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.477868    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw" podUID="76cef0b8-cc9d-4821-9456-e3189cf59ec5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:04:46.514406 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.514378    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:04:46.608596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.608567    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"]
Nov 01 23:04:46.629330 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.629310    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-w0-c54de0-kserve-6f9458768d-9cwzh"]
Nov 01 23:04:46.715488 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.715424    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"]
Nov 01 23:04:46.718109 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:46.718088    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/simulator-w0-c54de0-kserve-86b78d8c7b-tdlmd"]
Nov 01 23:04:48.524718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:48.524644    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" path="/var/lib/kubelet/pods/05a78acc-77a3-455d-9dd7-2e4f827fa20d/volumes"
Nov 01 23:04:48.525104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:48.525086    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" path="/var/lib/kubelet/pods/cb48bd1c-2d40-4e8e-93ae-a6735406208d/volumes"
Nov 01 23:04:48.813115 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:04:48.813008    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:04:56.477799 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:56.477763    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw" podUID="3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:04:56.478212 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:04:56.477771    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw" podUID="76cef0b8-cc9d-4821-9456-e3189cf59ec5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:05:04.904182 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:04.903821    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 23:05:04.997433 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:04.996606    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-kube-api-access-49g4v\" (UniqueName: \"kubernetes.io/projected/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-kube-api-access-49g4v\") pod \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\" (UID: \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\") "
Nov 01 23:05:04.997433 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:04.996739    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-plugins-config-volume\") pod \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\" (UID: \"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c\") "
Nov 01 23:05:04.997623 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:04.997452    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-plugins-config-volume" pod "3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" (UID: "3f99ea57-d150-4f65-b5e3-6c1ea645fd1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:05:04.998201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:04.998150    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:05.006150 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.006113    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-kube-api-access-49g4v" pod "3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" (UID: "3f99ea57-d150-4f65-b5e3-6c1ea645fd1c"). InnerVolumeSpecName "kube-api-access-49g4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:05.099811 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.099741    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49g4v\" (UniqueName: \"kubernetes.io/projected/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c-kube-api-access-49g4v\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:05.101073 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.101038    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 23:05:05.200489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.200460    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cef0b8-cc9d-4821-9456-e3189cf59ec5-kube-api-access-86wls\" (UniqueName: \"kubernetes.io/projected/76cef0b8-cc9d-4821-9456-e3189cf59ec5-kube-api-access-86wls\") pod \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\" (UID: \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\") "
Nov 01 23:05:05.200663 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.200500    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cef0b8-cc9d-4821-9456-e3189cf59ec5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cef0b8-cc9d-4821-9456-e3189cf59ec5-plugins-config-volume\") pod \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\" (UID: \"76cef0b8-cc9d-4821-9456-e3189cf59ec5\") "
Nov 01 23:05:05.201094 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.201059    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cef0b8-cc9d-4821-9456-e3189cf59ec5-plugins-config-volume" pod "76cef0b8-cc9d-4821-9456-e3189cf59ec5" (UID: "76cef0b8-cc9d-4821-9456-e3189cf59ec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:05:05.202560 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.202535    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cef0b8-cc9d-4821-9456-e3189cf59ec5-kube-api-access-86wls" pod "76cef0b8-cc9d-4821-9456-e3189cf59ec5" (UID: "76cef0b8-cc9d-4821-9456-e3189cf59ec5"). InnerVolumeSpecName "kube-api-access-86wls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:05.302222 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.302192    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86wls\" (UniqueName: \"kubernetes.io/projected/76cef0b8-cc9d-4821-9456-e3189cf59ec5-kube-api-access-86wls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:05.302222 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.302219    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cef0b8-cc9d-4821-9456-e3189cf59ec5-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:05.615175 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.615144    4837 generic.go:356] "Generic (PLEG): container finished" podID="76cef0b8-cc9d-4821-9456-e3189cf59ec5" containerID="c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9" exitCode=0
Nov 01 23:05:05.615352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.615194    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"
Nov 01 23:05:05.615352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.615231    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw" event={"ID":"76cef0b8-cc9d-4821-9456-e3189cf59ec5","Type":"ContainerDied","Data":"c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9"}
Nov 01 23:05:05.615352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.615294    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw" event={"ID":"76cef0b8-cc9d-4821-9456-e3189cf59ec5","Type":"ContainerDied","Data":"eb9bdea4457e2fc135ac594ec08702dae69ecc3a93f3387c8aa89960a9e203ac"}
Nov 01 23:05:05.615352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.615328    4837 scope.go:122] "RemoveContainer" containerID="c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9"
Nov 01 23:05:05.616806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.616777    4837 generic.go:356] "Generic (PLEG): container finished" podID="3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" containerID="9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147" exitCode=0
Nov 01 23:05:05.616982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.616962    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"
Nov 01 23:05:05.617105 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.616900    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw" event={"ID":"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c","Type":"ContainerDied","Data":"9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147"}
Nov 01 23:05:05.617178 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.617101    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw" event={"ID":"3f99ea57-d150-4f65-b5e3-6c1ea645fd1c","Type":"ContainerDied","Data":"f565711648aa8e7b1f544a2f4b636a2a3e9c9fde5fb0d284abf4ffc458df0fca"}
Nov 01 23:05:05.626217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.626199    4837 scope.go:122] "RemoveContainer" containerID="c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9"
Nov 01 23:05:05.626968 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:05.626935    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9\": container with ID starting with c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9 not found: ID does not exist" containerID="c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9"
Nov 01 23:05:05.627106 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.626978    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9"} err="failed to get container status \"c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9\": rpc error: code = NotFound desc = could not find container \"c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9\": container with ID starting with c978a05829374c042be7b6eb540de0f7111da9fad83766e920a02799fa51a9e9 not found: ID does not exist"
Nov 01 23:05:05.627384 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.627368    4837 scope.go:122] "RemoveContainer" containerID="9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147"
Nov 01 23:05:05.648606 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.648572    4837 scope.go:122] "RemoveContainer" containerID="9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147"
Nov 01 23:05:05.649005 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:05.648980    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147\": container with ID starting with 9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147 not found: ID does not exist" containerID="9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147"
Nov 01 23:05:05.649101 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.649026    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147"} err="failed to get container status \"9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147\": rpc error: code = NotFound desc = could not find container \"9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147\": container with ID starting with 9e40724547842e1be8a59da218f6d030b004073bfc3da55182f4f181205e2147 not found: ID does not exist"
Nov 01 23:05:05.771639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.771552    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"]
Nov 01 23:05:05.775845 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.775814    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c54de0-848dc78d69-r9psw"]
Nov 01 23:05:05.873396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.873361    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"]
Nov 01 23:05:05.878959 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:05.878932    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c54de0-685cc6f455-f49cw"]
Nov 01 23:05:06.526276 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:06.526231    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" path="/var/lib/kubelet/pods/3f99ea57-d150-4f65-b5e3-6c1ea645fd1c/volumes"
Nov 01 23:05:06.526829 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:06.526811    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cef0b8-cc9d-4821-9456-e3189cf59ec5" path="/var/lib/kubelet/pods/76cef0b8-cc9d-4821-9456-e3189cf59ec5/volumes"
Nov 01 23:05:09.610142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.610067    4837 patch_prober.go:29] interesting pod/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 23:05:09.610142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.610128    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 23:05:09.654322 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.654288    4837 generic.go:356] "Generic (PLEG): container finished" podID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerID="fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b" exitCode=0
Nov 01 23:05:09.654479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.654366    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" event={"ID":"83ebe69f-f0df-4388-b1d1-b15c1e43351e","Type":"ContainerDied","Data":"fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b"}
Nov 01 23:05:09.654795 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.654766    4837 scope.go:122] "RemoveContainer" containerID="fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b"
Nov 01 23:05:09.656293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.656272    4837 generic.go:356] "Generic (PLEG): container finished" podID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerID="c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6" exitCode=0
Nov 01 23:05:09.656402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.656314    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" event={"ID":"0165cb48-bf04-4815-a668-31a3af4f64ed","Type":"ContainerDied","Data":"c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6"}
Nov 01 23:05:09.656567 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.656538    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:05:09.656638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.656621    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:05:09.656683 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.656668    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:05:09.656766 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.656672    4837 scope.go:122] "RemoveContainer" containerID="c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6"
Nov 01 23:05:09.658136 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.658111    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:05:09.658207 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.658181    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:05:09.658245 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:09.658213    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:05:10.701637 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:10.701604    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" event={"ID":"0165cb48-bf04-4815-a668-31a3af4f64ed","Type":"ContainerStarted","Data":"edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df"}
Nov 01 23:05:10.703519 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:10.703493    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" event={"ID":"83ebe69f-f0df-4388-b1d1-b15c1e43351e","Type":"ContainerStarted","Data":"e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5"}
Nov 01 23:05:11.348068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:11.347448    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:05:11.348068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:11.347577    4837 patch_prober.go:29] interesting pod/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 23:05:11.348068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:11.347624    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 23:05:11.616928 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:11.616843    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:05:11.618277 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:11.618247    4837 patch_prober.go:29] interesting pod/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 01 23:05:11.618401 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:11.618302    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 01 23:05:12.344431 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:12.344392    4837 patch_prober.go:29] interesting pod/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 23:05:12.345076 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:12.345020    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 23:05:12.617271 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:12.617182    4837 patch_prober.go:29] interesting pod/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 01 23:05:12.617493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:12.617463    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 01 23:05:13.346925 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:13.346889    4837 patch_prober.go:29] interesting pod/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 23:05:13.347541 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:13.347512    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 23:05:13.617566 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:13.617487    4837 patch_prober.go:29] interesting pod/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 01 23:05:13.617566 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:13.617537    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 01 23:05:14.344413 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:14.344377    4837 patch_prober.go:29] interesting pod/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 23:05:14.344724 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:14.344695    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 23:05:14.617462 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:14.617374    4837 patch_prober.go:29] interesting pod/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 01 23:05:14.617462 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:14.617431    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 01 23:05:14.684596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:14.684561    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:05:15.344224 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:15.344187    4837 patch_prober.go:29] interesting pod/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 23:05:15.344526 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:15.344499    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 23:05:15.617594 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:15.617518    4837 patch_prober.go:29] interesting pod/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 01 23:05:15.617594 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:15.617579    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 01 23:05:15.776679 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:15.776644    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:05:16.348875 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:16.348844    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:05:16.349166 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:16.349145    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:05:16.349896 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:16.349873    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:05:16.622585 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:16.622510    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:05:16.748534 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:16.748502    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:05:18.813423 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:18.813392    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:05:40.929369 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.929338    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"]
Nov 01 23:05:40.929936 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.929921    4837 cpu_manager.go:414] "containerMap: removing container" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.929937    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.929955    4837 cpu_manager.go:414] "containerMap: removing container" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.929963    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.929975    4837 cpu_manager.go:414] "containerMap: removing container" podUID="a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" containerName="payload-processing"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.929983    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" containerName="payload-processing"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.929996    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" containerName="payload-processing"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930001    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" containerName="payload-processing"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930009    4837 cpu_manager.go:414] "containerMap: removing container" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy"
Nov 01 23:05:40.930011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930014    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930023    4837 cpu_manager.go:414] "containerMap: removing container" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930028    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930037    4837 cpu_manager.go:414] "containerMap: removing container" podUID="76cef0b8-cc9d-4821-9456-e3189cf59ec5" containerName="payload-pre-processing"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930057    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cef0b8-cc9d-4821-9456-e3189cf59ec5" containerName="payload-pre-processing"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930076    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930081    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930099    4837 cpu_manager.go:414] "containerMap: removing container" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930104    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930201    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930211    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930219    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ac5f8f-d582-4e2a-b94b-2f727121a5f7" containerName="payload-processing"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930227    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f99ea57-d150-4f65-b5e3-6c1ea645fd1c" containerName="payload-processing"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930233    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cddca7c6-47ca-440d-847e-1f1178de7051" containerName="istio-proxy"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930238    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a78acc-77a3-455d-9dd7-2e4f827fa20d" containerName="main"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930244    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cef0b8-cc9d-4821-9456-e3189cf59ec5" containerName="payload-pre-processing"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930251    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2661a9df-eef7-4201-bdb9-0f6311a11c88" containerName="istio-proxy"
Nov 01 23:05:40.930375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.930257    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb48bd1c-2d40-4e8e-93ae-a6735406208d" containerName="main"
Nov 01 23:05:40.935018 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.935001    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:40.976742 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.976718    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-metrics-certs\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:40.976883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.976802    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc50a1c4-0675-4939-8246-64c8d7cdbeec-openshift-service-ca\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:40.976974 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.976958    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29ms\" (UniqueName: \"kubernetes.io/projected/bc50a1c4-0675-4939-8246-64c8d7cdbeec-kube-api-access-v29ms\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:40.977059 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:40.976994    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.032900 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.032868    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3eb631-serving-cert\""
Nov 01 23:05:41.035682 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.035660    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"]
Nov 01 23:05:41.078158 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.078131    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-metrics-certs\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.078295 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.078172    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc50a1c4-0675-4939-8246-64c8d7cdbeec-openshift-service-ca\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.078400 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.078381    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29ms\" (UniqueName: \"kubernetes.io/projected/bc50a1c4-0675-4939-8246-64c8d7cdbeec-kube-api-access-v29ms\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.078480 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.078422    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.078597 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:41.078578    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-serving-cert: secret "maas-api-e2e-api-b-3eb631-serving-cert" not found
Nov 01 23:05:41.078760 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:41.078748    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls podName:bc50a1c4-0675-4939-8246-64c8d7cdbeec nodeName:}" failed. No retries permitted until 2026-11-01 23:05:41.578728683 +0000 UTC m=+2523.167927735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls") pod "maas-api-e2e-api-b-3eb631-887f9557d-78zj6" (UID: "bc50a1c4-0675-4939-8246-64c8d7cdbeec") : secret "maas-api-e2e-api-b-3eb631-serving-cert" not found
Nov 01 23:05:41.079100 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.079073    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc50a1c4-0675-4939-8246-64c8d7cdbeec-openshift-service-ca\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.080776 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.080756    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-metrics-certs\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.142787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.142759    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29ms\" (UniqueName: \"kubernetes.io/projected/bc50a1c4-0675-4939-8246-64c8d7cdbeec-kube-api-access-v29ms\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.588392 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.588358    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.590809 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.590785    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls\") pod \"maas-api-e2e-api-b-3eb631-887f9557d-78zj6\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:41.849440 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:41.849360    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:42.220435 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:42.220405    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"]
Nov 01 23:05:42.221222 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:05:42.221199    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc50a1c4_0675_4939_8246_64c8d7cdbeec.slice/crio-b630f1e83baa4827f73e2dc40dbc79d2ce3d4c5cba1199bbeffd37b2c102e086 WatchSource:0}: Error finding container b630f1e83baa4827f73e2dc40dbc79d2ce3d4c5cba1199bbeffd37b2c102e086: Status 404 returned error can't find the container with id b630f1e83baa4827f73e2dc40dbc79d2ce3d4c5cba1199bbeffd37b2c102e086
Nov 01 23:05:42.901775 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:42.901733    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6" event={"ID":"bc50a1c4-0675-4939-8246-64c8d7cdbeec","Type":"ContainerStarted","Data":"b630f1e83baa4827f73e2dc40dbc79d2ce3d4c5cba1199bbeffd37b2c102e086"}
Nov 01 23:05:43.909731 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:43.909652    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6" event={"ID":"bc50a1c4-0675-4939-8246-64c8d7cdbeec","Type":"ContainerStarted","Data":"d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582"}
Nov 01 23:05:43.910102 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:43.909771    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:44.021790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:44.021743    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6" podStartSLOduration=3.141850785 podStartE2EDuration="4.021728728s" totalImagesPullingTime="879.877943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:44.02002905 +0000 UTC m=+2525.609228110" watchObservedRunningTime="2026-11-01 23:05:44.021728728 +0000 UTC m=+2525.610927788"
Nov 01 23:05:48.813309 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:48.813276    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:05:49.922173 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:49.922147    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:05:50.083688 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:50.083648    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"]
Nov 01 23:05:50.083875 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:50.083853    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy" containerID="cri-o://3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3" gracePeriod=30
Nov 01 23:05:51.065271 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:51.065241    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"]
Nov 01 23:05:51.065700 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:51.065467    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy" containerID="cri-o://f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0" gracePeriod=30
Nov 01 23:05:52.572593 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:52.572559    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:05:52.572976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:52.572621    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:05:53.264981 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:53.264946    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:05:54.526612 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:54.526583    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"]
Nov 01 23:05:54.527031 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:54.526748    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy" containerID="cri-o://3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3" gracePeriod=30
Nov 01 23:05:55.463022 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.462992    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 23:05:55.491828 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.491797    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-envoy\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.491828 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.491831    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-kube-api-access-gt6rw\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-kube-api-access-gt6rw\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.492022 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.491879    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-socket\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.492022 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.491960    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-certs\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.492152 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492134    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-credential-socket\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.492209 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492170    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-token\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-token\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.492269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492241    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-podinfo\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.492269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492219    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-socket" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:55.492383 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492365    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d77d32-49a4-405f-af93-db6b4fc0e303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d77d32-49a4-405f-af93-db6b4fc0e303-istiod-ca-cert\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.492446 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492419    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-data\") pod \"70d77d32-49a4-405f-af93-db6b4fc0e303\" (UID: \"70d77d32-49a4-405f-af93-db6b4fc0e303\") "
Nov 01 23:05:55.492777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492411    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-credential-socket" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:55.492777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492626    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-certs" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:55.492777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492720    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d77d32-49a4-405f-af93-db6b4fc0e303-istiod-ca-cert" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:05:55.492777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.492750    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-data" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:55.493104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.493080    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.493214 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.493111    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.493214 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.493125    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d77d32-49a4-405f-af93-db6b4fc0e303-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.493214 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.493137    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.493214 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.493151    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.494371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.494347    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-envoy" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:55.494577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.494557    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-kube-api-access-gt6rw" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "kube-api-access-gt6rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:55.494643 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.494586    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-podinfo" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:05:55.495054 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.495027    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-token" pod "70d77d32-49a4-405f-af93-db6b4fc0e303" (UID: "70d77d32-49a4-405f-af93-db6b4fc0e303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:55.594084 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.594038    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.594084 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.594087    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6rw\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-kube-api-access-gt6rw\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.594541 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.594105    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.594541 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.594121    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d77d32-49a4-405f-af93-db6b4fc0e303-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:55.985149 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.985113    4837 generic.go:356] "Generic (PLEG): container finished" podID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerID="3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3" exitCode=0
Nov 01 23:05:55.985313 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.985157    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"
Nov 01 23:05:55.985313 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.985190    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" event={"ID":"70d77d32-49a4-405f-af93-db6b4fc0e303","Type":"ContainerDied","Data":"3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3"}
Nov 01 23:05:55.985313 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.985248    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92" event={"ID":"70d77d32-49a4-405f-af93-db6b4fc0e303","Type":"ContainerDied","Data":"b604b334671d789eb7135ad4bcdd5b2c127d1b035c6ade7b98089429da85aab5"}
Nov 01 23:05:55.985313 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.985271    4837 scope.go:122] "RemoveContainer" containerID="3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3"
Nov 01 23:05:55.997536 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:55.997519    4837 scope.go:122] "RemoveContainer" containerID="3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6"
Nov 01 23:05:56.008654 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.008326    4837 scope.go:122] "RemoveContainer" containerID="3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3"
Nov 01 23:05:56.008723 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:56.008646    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3\": container with ID starting with 3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3 not found: ID does not exist" containerID="3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3"
Nov 01 23:05:56.008723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.008677    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3"} err="failed to get container status \"3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3\": rpc error: code = NotFound desc = could not find container \"3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3\": container with ID starting with 3fd77f8d31b1267391cc7b988d986bfcc62b711a12845f0a12fe3360c5ba99c3 not found: ID does not exist"
Nov 01 23:05:56.008723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.008699    4837 scope.go:122] "RemoveContainer" containerID="3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6"
Nov 01 23:05:56.009347 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:56.009018    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6\": container with ID starting with 3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6 not found: ID does not exist" containerID="3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6"
Nov 01 23:05:56.009347 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.009079    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6"} err="failed to get container status \"3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6\": rpc error: code = NotFound desc = could not find container \"3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6\": container with ID starting with 3cf342595c7ac8addc53e282a971887f43334cc9af9b4084d538d8d36dde0ff6 not found: ID does not exist"
Nov 01 23:05:56.153863 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.153835    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"]
Nov 01 23:05:56.156755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.156714    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-hmm92"]
Nov 01 23:05:56.336636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.336613    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 23:05:56.400039 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400007    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-kube-api-access-c5mvc\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-kube-api-access-c5mvc\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400258 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400086    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-token\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-token\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400258 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400174    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-certs\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400258 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400204    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-podinfo\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400258 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400226    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-socket\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400284    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-credential-socket\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400342    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-envoy\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400420    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-data\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400442    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-certs" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.400511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400471    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b5a76c-198e-4891-8383-a92528dcc6ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b5a76c-198e-4891-8383-a92528dcc6ca-istiod-ca-cert\") pod \"85b5a76c-198e-4891-8383-a92528dcc6ca\" (UID: \"85b5a76c-198e-4891-8383-a92528dcc6ca\") "
Nov 01 23:05:56.400806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400629    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-credential-socket" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.400806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400763    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-data" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.400806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400770    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-socket" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.401005 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400964    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.401005 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.400985    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.401005 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.401001    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.401171 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.401036    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.401171 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.401148    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b5a76c-198e-4891-8383-a92528dcc6ca-istiod-ca-cert" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:05:56.402395 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.402373    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-kube-api-access-c5mvc" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "kube-api-access-c5mvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:56.402545 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.402531    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-podinfo" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:05:56.402755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.402732    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-envoy" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.402855 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.402765    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-token" pod "85b5a76c-198e-4891-8383-a92528dcc6ca" (UID: "85b5a76c-198e-4891-8383-a92528dcc6ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:56.502325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.502245    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.502325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.502278    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.502325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.502288    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b5a76c-198e-4891-8383-a92528dcc6ca-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.502325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.502297    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5mvc\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-kube-api-access-c5mvc\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.502325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.502313    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b5a76c-198e-4891-8383-a92528dcc6ca-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.525707 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.525674    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" path="/var/lib/kubelet/pods/70d77d32-49a4-405f-af93-db6b4fc0e303/volumes"
Nov 01 23:05:56.993997 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.993964    4837 generic.go:356] "Generic (PLEG): container finished" podID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerID="f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0" exitCode=0
Nov 01 23:05:56.994472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.994015    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"
Nov 01 23:05:56.994472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.994075    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" event={"ID":"85b5a76c-198e-4891-8383-a92528dcc6ca","Type":"ContainerDied","Data":"f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0"}
Nov 01 23:05:56.994472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.994137    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv" event={"ID":"85b5a76c-198e-4891-8383-a92528dcc6ca","Type":"ContainerDied","Data":"435b21988025a06a07e805a54bcb5b26903ca17e0735ce757288b096a68167fc"}
Nov 01 23:05:56.994472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:56.994169    4837 scope.go:122] "RemoveContainer" containerID="f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0"
Nov 01 23:05:57.004535 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:57.004518    4837 scope.go:122] "RemoveContainer" containerID="aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d"
Nov 01 23:05:57.013419 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:57.013398    4837 scope.go:122] "RemoveContainer" containerID="f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0"
Nov 01 23:05:57.013683 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:57.013662    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0\": container with ID starting with f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0 not found: ID does not exist" containerID="f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0"
Nov 01 23:05:57.013741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:57.013691    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0"} err="failed to get container status \"f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0\": rpc error: code = NotFound desc = could not find container \"f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0\": container with ID starting with f9d2bdf2952a8134c8242f4efbd7070189803292f8259d3fe9fb8167463a8ff0 not found: ID does not exist"
Nov 01 23:05:57.013741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:57.013712    4837 scope.go:122] "RemoveContainer" containerID="aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d"
Nov 01 23:05:57.013932 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:05:57.013917    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d\": container with ID starting with aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d not found: ID does not exist" containerID="aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d"
Nov 01 23:05:57.013980 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:57.013936    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d"} err="failed to get container status \"aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d\": rpc error: code = NotFound desc = could not find container \"aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d\": container with ID starting with aad608c4d22965c1a832fb73599dd878c3e4c0b16a18c339912fd107732be04d not found: ID does not exist"
Nov 01 23:05:57.149667 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:57.149635    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"]
Nov 01 23:05:57.158455 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:57.158430    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-qfknv"]
Nov 01 23:05:58.526558 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:58.526528    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" path="/var/lib/kubelet/pods/85b5a76c-198e-4891-8383-a92528dcc6ca/volumes"
Nov 01 23:05:59.793759 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.793703    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 23:05:59.836619 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.836590    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-token\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.836812 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.836635    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-socket\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.836812 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.836667    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-podinfo\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.836812 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.836715    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-data\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.836812 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.836781    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-envoy\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.837077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.836978    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-socket" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:59.837077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837016    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ad4a38-83eb-498d-b205-3b42607b5060-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad4a38-83eb-498d-b205-3b42607b5060-istiod-ca-cert\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.837213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837072    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-data" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:59.837301 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837276    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-credential-socket\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.837387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837355    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-kube-api-access-8289w\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-kube-api-access-8289w\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.837468 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837448    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-certs\") pod \"e9ad4a38-83eb-498d-b205-3b42607b5060\" (UID: \"e9ad4a38-83eb-498d-b205-3b42607b5060\") "
Nov 01 23:05:59.837929 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837300    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ad4a38-83eb-498d-b205-3b42607b5060-istiod-ca-cert" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:05:59.838143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837521    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-credential-socket" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:59.838272 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837796    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-certs" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:59.838394 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.837961    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:59.838527 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.838394    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:59.839295 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.839268    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-token" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:59.839474 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.839442    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-podinfo" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:05:59.839583 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.839504    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-envoy" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:59.839761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.839742    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-kube-api-access-8289w" pod "e9ad4a38-83eb-498d-b205-3b42607b5060" (UID: "e9ad4a38-83eb-498d-b205-3b42607b5060"). InnerVolumeSpecName "kube-api-access-8289w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:59.939559 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.939522    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:59.939559 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.939551    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:59.939751 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.939565    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:59.939751 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.939579    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:59.939751 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.939591    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad4a38-83eb-498d-b205-3b42607b5060-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:59.939751 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.939602    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad4a38-83eb-498d-b205-3b42607b5060-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:59.939751 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:05:59.939614    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8289w\" (UniqueName: \"kubernetes.io/projected/e9ad4a38-83eb-498d-b205-3b42607b5060-kube-api-access-8289w\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:00.016593 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.016559    4837 generic.go:356] "Generic (PLEG): container finished" podID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerID="3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3" exitCode=0
Nov 01 23:06:00.016776 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.016620    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"
Nov 01 23:06:00.016776 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.016646    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" event={"ID":"e9ad4a38-83eb-498d-b205-3b42607b5060","Type":"ContainerDied","Data":"3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3"}
Nov 01 23:06:00.016776 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.016741    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj" event={"ID":"e9ad4a38-83eb-498d-b205-3b42607b5060","Type":"ContainerDied","Data":"51d6dc394bcfb8f9333f605a14772edf4b1291ac2d0d8a77175efed5b41482a9"}
Nov 01 23:06:00.016776 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.016771    4837 scope.go:122] "RemoveContainer" containerID="3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3"
Nov 01 23:06:00.030005 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.029989    4837 scope.go:122] "RemoveContainer" containerID="3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3"
Nov 01 23:06:00.030317 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:06:00.030298    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3\": container with ID starting with 3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3 not found: ID does not exist" containerID="3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3"
Nov 01 23:06:00.030369 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.030323    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3"} err="failed to get container status \"3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3\": rpc error: code = NotFound desc = could not find container \"3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3\": container with ID starting with 3b0b1b47a77a5b660eab1c2fa3fac940d2d54e63c41948b634ede91750a638e3 not found: ID does not exist"
Nov 01 23:06:00.172012 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.171979    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"]
Nov 01 23:06:00.175811 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.175782    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-gqrxj"]
Nov 01 23:06:00.525250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:00.525177    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" path="/var/lib/kubelet/pods/e9ad4a38-83eb-498d-b205-3b42607b5060/volumes"
Nov 01 23:06:14.569930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:14.569887    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"]
Nov 01 23:06:14.570402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:14.570146    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy" containerID="cri-o://0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c" gracePeriod=30
Nov 01 23:06:14.830616 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:14.830542    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:06:18.812727 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:06:18.812644    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:06:19.826537 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.826513    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 23:06:19.891205 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891140    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-kube-api-access-wlmzs\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-kube-api-access-wlmzs\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.891205 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891185    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-data\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.891408 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891359    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-podinfo\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.891475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891418    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-certs\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.891475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891442    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-socket\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.891596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891465    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-data" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.891596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891568    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-credential-socket\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.891715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891648    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-envoy\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.891715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891683    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-socket" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.891715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891689    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istiod-ca-cert\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.891715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891703    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-certs" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.891945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891822    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-credential-socket" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.891945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.891836    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-token\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-token\") pod \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\" (UID: \"f7c49f6c-74f6-4a86-ae15-b56eb5355230\") "
Nov 01 23:06:19.892086 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.892001    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istiod-ca-cert" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:06:19.892439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.892414    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.892439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.892443    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.892621 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.892459    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.892621 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.892472    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.892621 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.892485    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.893466 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.893444    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-kube-api-access-wlmzs" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "kube-api-access-wlmzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:06:19.893579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.893485    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-podinfo" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:06:19.893759 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.893732    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-envoy" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.893897 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.893881    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-token" pod "f7c49f6c-74f6-4a86-ae15-b56eb5355230" (UID: "f7c49f6c-74f6-4a86-ae15-b56eb5355230"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:06:19.993489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.993458    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.993489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.993487    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.993653 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.993502    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlmzs\" (UniqueName: \"kubernetes.io/projected/f7c49f6c-74f6-4a86-ae15-b56eb5355230-kube-api-access-wlmzs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.993653 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:19.993517    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c49f6c-74f6-4a86-ae15-b56eb5355230-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:20.134975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.134943    4837 generic.go:356] "Generic (PLEG): container finished" podID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerID="0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c" exitCode=0
Nov 01 23:06:20.135162 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.134989    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"
Nov 01 23:06:20.135162 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.135009    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" event={"ID":"f7c49f6c-74f6-4a86-ae15-b56eb5355230","Type":"ContainerDied","Data":"0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c"}
Nov 01 23:06:20.135162 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.135063    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" event={"ID":"f7c49f6c-74f6-4a86-ae15-b56eb5355230","Type":"ContainerDied","Data":"140c9c689411399921f9ce074cc9cbbc6271691a9b5771f5dc040ff12a62b121"}
Nov 01 23:06:20.135162 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.135084    4837 scope.go:122] "RemoveContainer" containerID="0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c"
Nov 01 23:06:20.146291 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.146265    4837 scope.go:122] "RemoveContainer" containerID="0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c"
Nov 01 23:06:20.146543 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:06:20.146525    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c\": container with ID starting with 0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c not found: ID does not exist" containerID="0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c"
Nov 01 23:06:20.146613 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.146555    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c"} err="failed to get container status \"0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c\": rpc error: code = NotFound desc = could not find container \"0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c\": container with ID starting with 0a9330184b6ff25d0159850fb7ebfa03d7f7fbd9b340208acdd59a4612cc759c not found: ID does not exist"
Nov 01 23:06:20.290015 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.289987    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"]
Nov 01 23:06:20.294613 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.294590    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz"]
Nov 01 23:06:20.524306 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.524280    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" path="/var/lib/kubelet/pods/f7c49f6c-74f6-4a86-ae15-b56eb5355230/volumes"
Nov 01 23:06:20.795204 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.795124    4837 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:06:20.795204 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:20.795192    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-2h5hz" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:06:29.900445 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900411    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"]
Nov 01 23:06:29.900941 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900927    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy"
Nov 01 23:06:29.900941 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900943    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900958    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900963    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900975    4837 cpu_manager.go:414] "containerMap: removing container" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900980    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900987    4837 cpu_manager.go:414] "containerMap: removing container" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.900991    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901013    4837 cpu_manager.go:414] "containerMap: removing container" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901019    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901056    4837 cpu_manager.go:414] "containerMap: removing container" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901064    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901083    4837 cpu_manager.go:414] "containerMap: removing container" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy"
Nov 01 23:06:29.901168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901090    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy"
Nov 01 23:06:29.901638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901223    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy"
Nov 01 23:06:29.901638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901238    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy"
Nov 01 23:06:29.901638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901247    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ad4a38-83eb-498d-b205-3b42607b5060" containerName="istio-proxy"
Nov 01 23:06:29.901638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901256    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c49f6c-74f6-4a86-ae15-b56eb5355230" containerName="istio-proxy"
Nov 01 23:06:29.901638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901262    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b5a76c-198e-4891-8383-a92528dcc6ca" containerName="istio-proxy"
Nov 01 23:06:29.901638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.901272    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b56f6a2-6dad-405e-9d88-c30f0f9cee77" containerName="istio-proxy"
Nov 01 23:06:29.906331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:29.906315    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.001880 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.001842    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4af44d\"/\"openshift-service-ca.crt\""
Nov 01 23:06:30.001880 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.001863    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4af44d\"/\"kube-root-ca.crt\""
Nov 01 23:06:30.002091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.002030    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4af44d\"/\"test-model-4af44d-kserve-self-signed-certs\""
Nov 01 23:06:30.002091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.002031    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4af44d\"/\"default-dockercfg-2xchc\""
Nov 01 23:06:30.004722 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.004706    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"]
Nov 01 23:06:30.009709 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.009691    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b4e706-377c-43f8-ae38-8a610dcb0446-tls-certs\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.009810 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.009723    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-home\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.009891 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.009873    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-tmp-dir\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.009934 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.009902    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrn7\" (UniqueName: \"kubernetes.io/projected/34b4e706-377c-43f8-ae38-8a610dcb0446-kube-api-access-2zrn7\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.010007 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.009978    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-model-cache\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.010143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.010088    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-dshm\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.111591 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.111555    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-model-cache\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.111761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.111610    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-dshm\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.111761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.111649    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b4e706-377c-43f8-ae38-8a610dcb0446-tls-certs\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.111761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.111670    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-home\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.111933 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.111778    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-tmp-dir\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.111933 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.111805    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrn7\" (UniqueName: \"kubernetes.io/projected/34b4e706-377c-43f8-ae38-8a610dcb0446-kube-api-access-2zrn7\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.112306 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.112267    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-tmp-dir\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.112306 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.112293    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-home\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.112536 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.112293    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-model-cache\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.113930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.113910    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-dshm\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.114249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.114230    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b4e706-377c-43f8-ae38-8a610dcb0446-tls-certs\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.165499 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.165443    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrn7\" (UniqueName: \"kubernetes.io/projected/34b4e706-377c-43f8-ae38-8a610dcb0446-kube-api-access-2zrn7\") pod \"test-model-4af44d-kserve-6df7654965-ntwk7\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") " pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.219185 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.219157    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:30.584247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:30.584221    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"]
Nov 01 23:06:30.585741 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:06:30.585714    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34b4e706_377c_43f8_ae38_8a610dcb0446.slice/crio-671134c8af8225cce4835f58aeaba66e9ed4306961614cc13bb08281f15ef679 WatchSource:0}: Error finding container 671134c8af8225cce4835f58aeaba66e9ed4306961614cc13bb08281f15ef679: Status 404 returned error can't find the container with id 671134c8af8225cce4835f58aeaba66e9ed4306961614cc13bb08281f15ef679
Nov 01 23:06:31.198266 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:31.198235    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7" event={"ID":"34b4e706-377c-43f8-ae38-8a610dcb0446","Type":"ContainerStarted","Data":"ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7"}
Nov 01 23:06:31.198659 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:31.198284    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7" event={"ID":"34b4e706-377c-43f8-ae38-8a610dcb0446","Type":"ContainerStarted","Data":"671134c8af8225cce4835f58aeaba66e9ed4306961614cc13bb08281f15ef679"}
Nov 01 23:06:31.219811 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:31.219787    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:31.308648 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:31.308607    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7" podStartSLOduration=2.308594394 podStartE2EDuration="2.308594394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:31.30701914 +0000 UTC m=+2572.896218200" watchObservedRunningTime="2026-11-01 23:06:31.308594394 +0000 UTC m=+2572.897793456"
Nov 01 23:06:36.631155 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.631119    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"]
Nov 01 23:06:36.631788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.631769    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d77d32-49a4-405f-af93-db6b4fc0e303" containerName="istio-proxy"
Nov 01 23:06:36.634692 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.634675    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.683579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.683549    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-aa47d8\"/\"openshift-service-ca.crt\""
Nov 01 23:06:36.688354 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.688321    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6js\" (UniqueName: \"kubernetes.io/projected/1236add6-9446-4069-b520-18789338dd10-kube-api-access-6d6js\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.688467 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.688387    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1236add6-9446-4069-b520-18789338dd10-tls-certs\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.688531 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.688512    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-dshm\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.688570 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.688561    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-model-cache\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.688603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.688585    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-home\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.688716 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.688695    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-tmp-dir\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.730074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.730031    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-aa47d8\"/\"kube-root-ca.crt\""
Nov 01 23:06:36.730231 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.730186    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-aa47d8\"/\"default-dockercfg-jgn69\""
Nov 01 23:06:36.730409 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.730390    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-aa47d8\"/\"test-model-aa47d8-kserve-self-signed-certs\""
Nov 01 23:06:36.733839 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.733817    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"]
Nov 01 23:06:36.796893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.792924    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6js\" (UniqueName: \"kubernetes.io/projected/1236add6-9446-4069-b520-18789338dd10-kube-api-access-6d6js\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.796893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.793082    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1236add6-9446-4069-b520-18789338dd10-tls-certs\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.796893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.793224    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-dshm\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.796893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.793286    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-model-cache\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.796893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.793315    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-home\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.796893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.793534    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-tmp-dir\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.813063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.802266    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-tmp-dir\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.813063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.806865    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-model-cache\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.823639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.823539    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-home\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.828582 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.826917    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-dshm\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.838000 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.837592    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1236add6-9446-4069-b520-18789338dd10-tls-certs\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.856553 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.856529    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6js\" (UniqueName: \"kubernetes.io/projected/1236add6-9446-4069-b520-18789338dd10-kube-api-access-6d6js\") pod \"test-model-aa47d8-kserve-bb55cdfb5-dxhkr\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") " pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:36.947389 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:36.947364    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:37.319883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:37.319855    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"]
Nov 01 23:06:37.320783 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:06:37.320758    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1236add6_9446_4069_b520_18789338dd10.slice/crio-f5da60e4f34bc81e833293cf86827fa42b07750df44390d55c3eeebe169f3c9b WatchSource:0}: Error finding container f5da60e4f34bc81e833293cf86827fa42b07750df44390d55c3eeebe169f3c9b: Status 404 returned error can't find the container with id f5da60e4f34bc81e833293cf86827fa42b07750df44390d55c3eeebe169f3c9b
Nov 01 23:06:38.243006 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:38.242966    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr" event={"ID":"1236add6-9446-4069-b520-18789338dd10","Type":"ContainerStarted","Data":"42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2"}
Nov 01 23:06:38.243414 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:38.243010    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr" event={"ID":"1236add6-9446-4069-b520-18789338dd10","Type":"ContainerStarted","Data":"f5da60e4f34bc81e833293cf86827fa42b07750df44390d55c3eeebe169f3c9b"}
Nov 01 23:06:38.352553 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:38.352495    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr" podStartSLOduration=2.352480323 podStartE2EDuration="2.352480323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:38.350448467 +0000 UTC m=+2579.939647528" watchObservedRunningTime="2026-11-01 23:06:38.352480323 +0000 UTC m=+2579.941679384"
Nov 01 23:06:38.948218 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:38.948191    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:40.034761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:40.034728    4837 scope.go:122] "RemoveContainer" containerID="4f34ddccdfabd8a81c95906e13e951ef21e89c6c34721aa8e30ac09f2476e75a"
Nov 01 23:06:40.219325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:40.219293    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:40.232623 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:40.232599    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:40.268836 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:40.268816    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:06:46.948498 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:46.948465    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:46.961103 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:46.961081    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:47.313558 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:47.313476    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:06:48.813114 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:06:48.813077    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:06:59.358935 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:06:59.358902    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:07:18.811995 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:07:18.811955    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:07:37.060997 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.060957    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"]
Nov 01 23:07:37.061634 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.061242    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" containerID="cri-o://f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3" gracePeriod=30
Nov 01 23:07:37.107465 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.107430    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"]
Nov 01 23:07:37.107758 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.107730    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" containerID="cri-o://8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1" gracePeriod=30
Nov 01 23:07:37.109885 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.109856    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"]
Nov 01 23:07:37.110073 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.110030    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" containerID="cri-o://db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7" gracePeriod=30
Nov 01 23:07:37.154405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.154367    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"]
Nov 01 23:07:37.154686 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.154657    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" containerID="cri-o://6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81" gracePeriod=30
Nov 01 23:07:37.636915 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.636876    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hdrlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:07:37.637122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.636958    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:07:37.651141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.651111    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hgg9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:07:37.651283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:37.651172    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:07:38.625371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:38.625340    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:07:41.428247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:41.428210    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4bhlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:07:41.428627 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:41.428269    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:07:41.552543 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:41.552516    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pjdqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:07:41.552697 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:41.552563    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:07:42.427321 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.427295    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 23:07:42.472019 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.471992    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 23:07:42.497775 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.497753    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 23:07:42.511383 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.511363    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 23:07:42.537200 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537175    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/828731a6-88ec-40b3-ac92-e57431e8dab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828731a6-88ec-40b3-ac92-e57431e8dab8-istiod-ca-cert\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.537382 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537364    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-certs\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.537463 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537446    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-credential-socket\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.537519 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537498    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-envoy\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.537579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537536    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828731a6-88ec-40b3-ac92-e57431e8dab8-istiod-ca-cert" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:07:42.537579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537543    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-socket\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.537695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537593    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-token\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-token\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.537695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537620    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-certs" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.537695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537637    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-podinfo\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.537859 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537695    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-data\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.537859 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537720    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-credential-socket" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.537859 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537749    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-kube-api-access-6xndr\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-kube-api-access-6xndr\") pod \"828731a6-88ec-40b3-ac92-e57431e8dab8\" (UID: \"828731a6-88ec-40b3-ac92-e57431e8dab8\") "
Nov 01 23:07:42.538033 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537877    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-socket" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.538033 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.537999    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-data" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.538244 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.538223    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.538305 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.538249    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.538305 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.538264    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.538305 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.538278    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828731a6-88ec-40b3-ac92-e57431e8dab8-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.538305 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.538291    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.539832 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.539809    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-podinfo" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:07:42.539832 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.539820    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-token" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:42.540156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.540134    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-envoy" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.540209 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.540161    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-kube-api-access-6xndr" pod "828731a6-88ec-40b3-ac92-e57431e8dab8" (UID: "828731a6-88ec-40b3-ac92-e57431e8dab8"). InnerVolumeSpecName "kube-api-access-6xndr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:42.639151 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639071    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-certs\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.639151 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639110    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19ac7195-82b7-4b1a-99de-21e23225824b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ac7195-82b7-4b1a-99de-21e23225824b-istio-podinfo\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.639151 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639145    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-kube-api-access-v4k88\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-kube-api-access-v4k88\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.639428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639176    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-kube-api-access-8t5px\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-kube-api-access-8t5px\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.639428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639204    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-credential-socket\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.639428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639241    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-envoy\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.639428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639285    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-data\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.639428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639283    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-certs" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.639428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639363    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-credential-socket\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.639428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639407    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-certs\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639437    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-podinfo\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639497    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-socket\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639538    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-data" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639547    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-credential-socket" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639548    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629d787b-d1d4-4102-90dc-2c095dd59a56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629d787b-d1d4-4102-90dc-2c095dd59a56-istiod-ca-cert\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639614    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-kube-api-access-qqlhk\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-kube-api-access-qqlhk\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639693    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-socket\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639734    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-data\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639742    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-credential-socket" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.639816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639765    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-certs\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.640385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639840    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629d787b-d1d4-4102-90dc-2c095dd59a56-istiod-ca-cert" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:07:42.640385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639879    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-certs" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.640385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.639988    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-certs" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.640385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640258    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-data" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.640385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640218    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-socket" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.640385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640292    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-socket" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640388    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-envoy\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640419    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-token\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-token\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640454    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-istio-token\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-istio-token\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640500    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-podinfo\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640537    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-data\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640568    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istiod-ca-cert\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640611    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-token\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-token\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640649    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-socket\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.640726 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640692    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-credential-socket\") pod \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\" (UID: \"e4835c74-5ce8-4b54-8c88-49aa8b706df0\") "
Nov 01 23:07:42.641235 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640802    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-envoy\") pod \"629d787b-d1d4-4102-90dc-2c095dd59a56\" (UID: \"629d787b-d1d4-4102-90dc-2c095dd59a56\") "
Nov 01 23:07:42.641235 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.640837    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ac7195-82b7-4b1a-99de-21e23225824b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ac7195-82b7-4b1a-99de-21e23225824b-istiod-ca-cert\") pod \"19ac7195-82b7-4b1a-99de-21e23225824b\" (UID: \"19ac7195-82b7-4b1a-99de-21e23225824b\") "
Nov 01 23:07:42.641389 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641333    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istiod-ca-cert" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:07:42.641562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641537    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xndr\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-kube-api-access-6xndr\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641568    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641585    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641600    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641616    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641635    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641649    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629d787b-d1d4-4102-90dc-2c095dd59a56-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641664    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641679    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641694    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641708    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641723    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641737    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641753    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828731a6-88ec-40b3-ac92-e57431e8dab8-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641768    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.641858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641806    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-kube-api-access-8t5px" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "kube-api-access-8t5px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:42.642561 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.641998    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ac7195-82b7-4b1a-99de-21e23225824b-istiod-ca-cert" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:07:42.642561 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.642231    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-data" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.642561 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.642271    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-credential-socket" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.642561 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.642430    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-socket" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.642883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.642858    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-kube-api-access-v4k88" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "kube-api-access-v4k88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:42.642973 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.642952    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-podinfo" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:07:42.643340 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.643317    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-envoy" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.643580 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.643559    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ac7195-82b7-4b1a-99de-21e23225824b-istio-podinfo" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:07:42.644032 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.643998    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-kube-api-access-qqlhk" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "kube-api-access-qqlhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:42.644238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.644211    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-token" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:42.644438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.644410    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-envoy" pod "e4835c74-5ce8-4b54-8c88-49aa8b706df0" (UID: "e4835c74-5ce8-4b54-8c88-49aa8b706df0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.644524 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.644493    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-istio-token" pod "19ac7195-82b7-4b1a-99de-21e23225824b" (UID: "19ac7195-82b7-4b1a-99de-21e23225824b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:42.645349 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.645331    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-envoy" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:42.645935 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.645915    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-podinfo" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:07:42.645994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.645931    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-token" pod "629d787b-d1d4-4102-90dc-2c095dd59a56" (UID: "629d787b-d1d4-4102-90dc-2c095dd59a56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:42.703358 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.703329    4837 generic.go:356] "Generic (PLEG): container finished" podID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerID="6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81" exitCode=0
Nov 01 23:07:42.703762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.703397    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"
Nov 01 23:07:42.703762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.703423    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" event={"ID":"e4835c74-5ce8-4b54-8c88-49aa8b706df0","Type":"ContainerDied","Data":"6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81"}
Nov 01 23:07:42.703762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.703495    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk" event={"ID":"e4835c74-5ce8-4b54-8c88-49aa8b706df0","Type":"ContainerDied","Data":"abe362a3128b484f3beae0d4e8f2f3444238d1e18f6dec6cb9de4c8f035a3fa1"}
Nov 01 23:07:42.703762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.703529    4837 scope.go:122] "RemoveContainer" containerID="6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81"
Nov 01 23:07:42.705147 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.705120    4837 generic.go:356] "Generic (PLEG): container finished" podID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerID="f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3" exitCode=0
Nov 01 23:07:42.705262 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.705190    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"
Nov 01 23:07:42.705262 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.705214    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" event={"ID":"828731a6-88ec-40b3-ac92-e57431e8dab8","Type":"ContainerDied","Data":"f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3"}
Nov 01 23:07:42.705472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.705261    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r" event={"ID":"828731a6-88ec-40b3-ac92-e57431e8dab8","Type":"ContainerDied","Data":"6ef617ce77400ace34f4038c553b1b51e3f15d4b2ec8d5d0d18f0b3d3d1ef1e1"}
Nov 01 23:07:42.712077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.709263    4837 generic.go:356] "Generic (PLEG): container finished" podID="19ac7195-82b7-4b1a-99de-21e23225824b" containerID="8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1" exitCode=0
Nov 01 23:07:42.712077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.709316    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"
Nov 01 23:07:42.712077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.709344    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" event={"ID":"19ac7195-82b7-4b1a-99de-21e23225824b","Type":"ContainerDied","Data":"8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1"}
Nov 01 23:07:42.712077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.709391    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk" event={"ID":"19ac7195-82b7-4b1a-99de-21e23225824b","Type":"ContainerDied","Data":"1220b04bc7f7448233e5820fcedf143555e40400635cd871371f2020fc535aeb"}
Nov 01 23:07:42.712077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.711163    4837 generic.go:356] "Generic (PLEG): container finished" podID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerID="db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7" exitCode=0
Nov 01 23:07:42.712077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.711196    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" event={"ID":"629d787b-d1d4-4102-90dc-2c095dd59a56","Type":"ContainerDied","Data":"db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7"}
Nov 01 23:07:42.712077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.711201    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"
Nov 01 23:07:42.712077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.711229    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj" event={"ID":"629d787b-d1d4-4102-90dc-2c095dd59a56","Type":"ContainerDied","Data":"8cbe377aaeede55713bf20ee1c13033c2eba9526c0448389b94ac2352eb59dc6"}
Nov 01 23:07:42.723341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.723258    4837 scope.go:122] "RemoveContainer" containerID="6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81"
Nov 01 23:07:42.723709 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:07:42.723690    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81\": container with ID starting with 6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81 not found: ID does not exist" containerID="6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81"
Nov 01 23:07:42.723803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.723715    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81"} err="failed to get container status \"6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81\": rpc error: code = NotFound desc = could not find container \"6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81\": container with ID starting with 6f4120ac9609cddccfb490673bcfc3084865930be785719b6ede89169d173f81 not found: ID does not exist"
Nov 01 23:07:42.723803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.723730    4837 scope.go:122] "RemoveContainer" containerID="f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3"
Nov 01 23:07:42.733268 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.733242    4837 scope.go:122] "RemoveContainer" containerID="f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3"
Nov 01 23:07:42.733551 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:07:42.733529    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3\": container with ID starting with f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3 not found: ID does not exist" containerID="f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3"
Nov 01 23:07:42.733626 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.733560    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3"} err="failed to get container status \"f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3\": rpc error: code = NotFound desc = could not find container \"f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3\": container with ID starting with f0cdb03958640c290b1e7f6a6588da44065a5c9432c2df449d56a2d346797df3 not found: ID does not exist"
Nov 01 23:07:42.733626 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.733584    4837 scope.go:122] "RemoveContainer" containerID="8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1"
Nov 01 23:07:42.742659 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.742637    4837 scope.go:122] "RemoveContainer" containerID="8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1"
Nov 01 23:07:42.742956 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:07:42.742938    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1\": container with ID starting with 8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1 not found: ID does not exist" containerID="8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1"
Nov 01 23:07:42.743028 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.742961    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1"} err="failed to get container status \"8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1\": rpc error: code = NotFound desc = could not find container \"8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1\": container with ID starting with 8c9b9358dcf7489d8da2076168e779212a4d900465d657e9b20970b8f236cae1 not found: ID does not exist"
Nov 01 23:07:42.743028 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.742977    4837 scope.go:122] "RemoveContainer" containerID="db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7"
Nov 01 23:07:42.743349 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743330    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743423 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743355    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqlhk\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-kube-api-access-qqlhk\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743423 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743372    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743423 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743387    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743423 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743400    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ac7195-82b7-4b1a-99de-21e23225824b-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743423 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743414    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743427    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743441    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743456    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ac7195-82b7-4b1a-99de-21e23225824b-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743474    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743487    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629d787b-d1d4-4102-90dc-2c095dd59a56-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743500    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ac7195-82b7-4b1a-99de-21e23225824b-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743514    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ac7195-82b7-4b1a-99de-21e23225824b-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743528    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4k88\" (UniqueName: \"kubernetes.io/projected/e4835c74-5ce8-4b54-8c88-49aa8b706df0-kube-api-access-v4k88\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743544    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5px\" (UniqueName: \"kubernetes.io/projected/629d787b-d1d4-4102-90dc-2c095dd59a56-kube-api-access-8t5px\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.743656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.743558    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4835c74-5ce8-4b54-8c88-49aa8b706df0-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:42.751634 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.751614    4837 scope.go:122] "RemoveContainer" containerID="db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7"
Nov 01 23:07:42.751923 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:07:42.751905    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7\": container with ID starting with db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7 not found: ID does not exist" containerID="db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7"
Nov 01 23:07:42.751999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:42.751926    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7"} err="failed to get container status \"db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7\": rpc error: code = NotFound desc = could not find container \"db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7\": container with ID starting with db4873296aeeda7b9c75b1a57f877e9ea73c5fafdf936ac4134a91531eee11a7 not found: ID does not exist"
Nov 01 23:07:43.066703 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:43.066674    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"]
Nov 01 23:07:43.076854 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:43.076828    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4bhlj"]
Nov 01 23:07:43.179562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:43.179534    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"]
Nov 01 23:07:43.184456 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:43.184433    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pjdqk"]
Nov 01 23:07:43.291807 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:43.291782    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"]
Nov 01 23:07:43.298260 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:43.298239    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hdrlk"]
Nov 01 23:07:43.404327 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:43.404271    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"]
Nov 01 23:07:43.415248 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:43.415226    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hgg9r"]
Nov 01 23:07:44.525012 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:44.524981    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" path="/var/lib/kubelet/pods/19ac7195-82b7-4b1a-99de-21e23225824b/volumes"
Nov 01 23:07:44.525522 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:44.525505    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" path="/var/lib/kubelet/pods/629d787b-d1d4-4102-90dc-2c095dd59a56/volumes"
Nov 01 23:07:44.525938 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:44.525910    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" path="/var/lib/kubelet/pods/828731a6-88ec-40b3-ac92-e57431e8dab8/volumes"
Nov 01 23:07:44.526655 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:07:44.526633    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" path="/var/lib/kubelet/pods/e4835c74-5ce8-4b54-8c88-49aa8b706df0/volumes"
Nov 01 23:07:48.814201 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:07:48.814118    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:08:13.713066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:08:13.713011    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:08:18.813198 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:08:18.813160    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:08:38.929063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:08:38.929018    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:08:38.931068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:08:38.931034    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:08:48.812021 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:08:48.811987    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:09:09.545550 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:09:09.545515    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:09:18.812718 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:09:18.812642    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:09:44.599688 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:09:44.599655    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:09:48.813224 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:09:48.813165    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:10:18.818449 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:10:18.818410    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:10:30.235364 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:30.235330    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:10:34.288254 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:34.288222    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"]
Nov 01 23:10:43.983468 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:43.983433    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"]
Nov 01 23:10:43.983992 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:43.983650    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b" podUID="9358a5c7-3313-4dee-b5e4-d0ac7a810fde" containerName="maas-api" containerID="cri-o://c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86" gracePeriod=30
Nov 01 23:10:44.267953 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.267928    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:10:44.440065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.440012    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-metrics-certs\") pod \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") "
Nov 01 23:10:44.440250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.440130    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-openshift-service-ca\") pod \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") "
Nov 01 23:10:44.440250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.440211    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls\") pod \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") "
Nov 01 23:10:44.440368 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.440253    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-kube-api-access-l4c45\" (UniqueName: \"kubernetes.io/projected/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-kube-api-access-l4c45\") pod \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\" (UID: \"9358a5c7-3313-4dee-b5e4-d0ac7a810fde\") "
Nov 01 23:10:44.440571 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.440548    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-openshift-service-ca" pod "9358a5c7-3313-4dee-b5e4-d0ac7a810fde" (UID: "9358a5c7-3313-4dee-b5e4-d0ac7a810fde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:10:44.440666 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.440634    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:44.442225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.442195    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-metrics-certs" pod "9358a5c7-3313-4dee-b5e4-d0ac7a810fde" (UID: "9358a5c7-3313-4dee-b5e4-d0ac7a810fde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:10:44.442332 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.442284    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls" pod "9358a5c7-3313-4dee-b5e4-d0ac7a810fde" (UID: "9358a5c7-3313-4dee-b5e4-d0ac7a810fde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:10:44.442332 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.442305    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-kube-api-access-l4c45" pod "9358a5c7-3313-4dee-b5e4-d0ac7a810fde" (UID: "9358a5c7-3313-4dee-b5e4-d0ac7a810fde"). InnerVolumeSpecName "kube-api-access-l4c45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:44.553573 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.553546    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4c45\" (UniqueName: \"kubernetes.io/projected/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-kube-api-access-l4c45\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:44.553781 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.553765    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:44.553903 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.553888    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9358a5c7-3313-4dee-b5e4-d0ac7a810fde-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:44.733275 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.733237    4837 generic.go:356] "Generic (PLEG): container finished" podID="9358a5c7-3313-4dee-b5e4-d0ac7a810fde" containerID="c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86" exitCode=0
Nov 01 23:10:44.733476 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.733313    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"
Nov 01 23:10:44.733476 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.733332    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b" event={"ID":"9358a5c7-3313-4dee-b5e4-d0ac7a810fde","Type":"ContainerDied","Data":"c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86"}
Nov 01 23:10:44.733476 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.733401    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b" event={"ID":"9358a5c7-3313-4dee-b5e4-d0ac7a810fde","Type":"ContainerDied","Data":"a039ea9aeb00a3b93b7aa77f88ce7ad4ad3813a15e5f2bf67fd06d6b744bb571"}
Nov 01 23:10:44.733476 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.733431    4837 scope.go:122] "RemoveContainer" containerID="c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86"
Nov 01 23:10:44.742901 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.742877    4837 scope.go:122] "RemoveContainer" containerID="c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86"
Nov 01 23:10:44.743192 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:10:44.743173    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86\": container with ID starting with c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86 not found: ID does not exist" containerID="c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86"
Nov 01 23:10:44.743276 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.743203    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86"} err="failed to get container status \"c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86\": rpc error: code = NotFound desc = could not find container \"c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86\": container with ID starting with c17512709cf6d31e848bf92bbafd650e798ac28b331b989e78cab6f9ebbddd86 not found: ID does not exist"
Nov 01 23:10:44.890903 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.890828    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"]
Nov 01 23:10:44.927871 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:44.927842    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-675d1d-f57df48d7-ftn9b"]
Nov 01 23:10:46.524479 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:46.524446    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9358a5c7-3313-4dee-b5e4-d0ac7a810fde" path="/var/lib/kubelet/pods/9358a5c7-3313-4dee-b5e4-d0ac7a810fde/volumes"
Nov 01 23:10:48.812339 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:10:48.812263    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:10:49.308034 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.307996    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7" podUID="34b4e706-377c-43f8-ae38-8a610dcb0446" containerName="main" containerID="cri-o://ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7" gracePeriod=45
Nov 01 23:10:49.575839 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.575785    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:10:49.608909 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.608825    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-dshm\") pod \"34b4e706-377c-43f8-ae38-8a610dcb0446\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") "
Nov 01 23:10:49.609065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.608948    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-home\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-home\") pod \"34b4e706-377c-43f8-ae38-8a610dcb0446\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") "
Nov 01 23:10:49.609065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609023    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b4e706-377c-43f8-ae38-8a610dcb0446-tls-certs\" (UniqueName: \"kubernetes.io/secret/34b4e706-377c-43f8-ae38-8a610dcb0446-tls-certs\") pod \"34b4e706-377c-43f8-ae38-8a610dcb0446\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") "
Nov 01 23:10:49.609206 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609081    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b4e706-377c-43f8-ae38-8a610dcb0446-kube-api-access-2zrn7\" (UniqueName: \"kubernetes.io/projected/34b4e706-377c-43f8-ae38-8a610dcb0446-kube-api-access-2zrn7\") pod \"34b4e706-377c-43f8-ae38-8a610dcb0446\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") "
Nov 01 23:10:49.609206 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609188    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-model-cache\") pod \"34b4e706-377c-43f8-ae38-8a610dcb0446\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") "
Nov 01 23:10:49.609313 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609229    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-home" pod "34b4e706-377c-43f8-ae38-8a610dcb0446" (UID: "34b4e706-377c-43f8-ae38-8a610dcb0446"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:49.609313 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609245    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-tmp-dir\") pod \"34b4e706-377c-43f8-ae38-8a610dcb0446\" (UID: \"34b4e706-377c-43f8-ae38-8a610dcb0446\") "
Nov 01 23:10:49.609426 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609394    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-model-cache" pod "34b4e706-377c-43f8-ae38-8a610dcb0446" (UID: "34b4e706-377c-43f8-ae38-8a610dcb0446"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:49.609555 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609522    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-tmp-dir" pod "34b4e706-377c-43f8-ae38-8a610dcb0446" (UID: "34b4e706-377c-43f8-ae38-8a610dcb0446"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:49.610161 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609913    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:49.610161 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609940    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:49.610161 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.609957    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:49.611100 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.611079    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-dshm" pod "34b4e706-377c-43f8-ae38-8a610dcb0446" (UID: "34b4e706-377c-43f8-ae38-8a610dcb0446"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:49.611300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.611274    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b4e706-377c-43f8-ae38-8a610dcb0446-kube-api-access-2zrn7" pod "34b4e706-377c-43f8-ae38-8a610dcb0446" (UID: "34b4e706-377c-43f8-ae38-8a610dcb0446"). InnerVolumeSpecName "kube-api-access-2zrn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:49.611388 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.611324    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b4e706-377c-43f8-ae38-8a610dcb0446-tls-certs" pod "34b4e706-377c-43f8-ae38-8a610dcb0446" (UID: "34b4e706-377c-43f8-ae38-8a610dcb0446"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:10:49.711800 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.711772    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34b4e706-377c-43f8-ae38-8a610dcb0446-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:49.711800 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.711804    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34b4e706-377c-43f8-ae38-8a610dcb0446-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:49.711959 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.711819    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zrn7\" (UniqueName: \"kubernetes.io/projected/34b4e706-377c-43f8-ae38-8a610dcb0446-kube-api-access-2zrn7\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:49.771754 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.771724    4837 generic.go:356] "Generic (PLEG): container finished" podID="34b4e706-377c-43f8-ae38-8a610dcb0446" containerID="ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7" exitCode=0
Nov 01 23:10:49.771912 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.771785    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"
Nov 01 23:10:49.771912 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.771802    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7" event={"ID":"34b4e706-377c-43f8-ae38-8a610dcb0446","Type":"ContainerDied","Data":"ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7"}
Nov 01 23:10:49.771912 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.771847    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7" event={"ID":"34b4e706-377c-43f8-ae38-8a610dcb0446","Type":"ContainerDied","Data":"671134c8af8225cce4835f58aeaba66e9ed4306961614cc13bb08281f15ef679"}
Nov 01 23:10:49.771912 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.771865    4837 scope.go:122] "RemoveContainer" containerID="ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7"
Nov 01 23:10:49.781870 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.781847    4837 scope.go:122] "RemoveContainer" containerID="ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7"
Nov 01 23:10:49.782172 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:10:49.782153    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7\": container with ID starting with ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7 not found: ID does not exist" containerID="ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7"
Nov 01 23:10:49.782253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.782181    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7"} err="failed to get container status \"ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7\": rpc error: code = NotFound desc = could not find container \"ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7\": container with ID starting with ff99871de686c59e341881d8d47b4b7fe5dadd5f4b60551db49f9b39666302f7 not found: ID does not exist"
Nov 01 23:10:49.924961 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.924936    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"]
Nov 01 23:10:49.928672 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:49.928643    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4af44d/test-model-4af44d-kserve-6df7654965-ntwk7"]
Nov 01 23:10:50.523892 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:50.523860    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b4e706-377c-43f8-ae38-8a610dcb0446" path="/var/lib/kubelet/pods/34b4e706-377c-43f8-ae38-8a610dcb0446/volumes"
Nov 01 23:10:55.727112 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:55.727082    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"]
Nov 01 23:10:55.727539 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:10:55.727299    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy" containerID="cri-o://ade887165161bcd91e2d20d1fe660aff4cbb7a7d556932171ad3ab396aaefebe" gracePeriod=30
Nov 01 23:11:00.268937 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.268906    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"]
Nov 01 23:11:00.269351 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.269124    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6" podUID="bc50a1c4-0675-4939-8246-64c8d7cdbeec" containerName="maas-api" containerID="cri-o://d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582" gracePeriod=30
Nov 01 23:11:00.568548 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.568520    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:11:00.626342 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.626311    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-metrics-certs\") pod \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") "
Nov 01 23:11:00.626535 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.626350    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls\") pod \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") "
Nov 01 23:11:00.626535 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.626387    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc50a1c4-0675-4939-8246-64c8d7cdbeec-kube-api-access-v29ms\" (UniqueName: \"kubernetes.io/projected/bc50a1c4-0675-4939-8246-64c8d7cdbeec-kube-api-access-v29ms\") pod \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") "
Nov 01 23:11:00.626679 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.626610    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc50a1c4-0675-4939-8246-64c8d7cdbeec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc50a1c4-0675-4939-8246-64c8d7cdbeec-openshift-service-ca\") pod \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\" (UID: \"bc50a1c4-0675-4939-8246-64c8d7cdbeec\") "
Nov 01 23:11:00.627122 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.627099    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc50a1c4-0675-4939-8246-64c8d7cdbeec-openshift-service-ca" pod "bc50a1c4-0675-4939-8246-64c8d7cdbeec" (UID: "bc50a1c4-0675-4939-8246-64c8d7cdbeec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:00.628763 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.628741    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls" pod "bc50a1c4-0675-4939-8246-64c8d7cdbeec" (UID: "bc50a1c4-0675-4939-8246-64c8d7cdbeec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:00.628763 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.628751    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc50a1c4-0675-4939-8246-64c8d7cdbeec-kube-api-access-v29ms" pod "bc50a1c4-0675-4939-8246-64c8d7cdbeec" (UID: "bc50a1c4-0675-4939-8246-64c8d7cdbeec"). InnerVolumeSpecName "kube-api-access-v29ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:00.628889 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.628828    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-metrics-certs" pod "bc50a1c4-0675-4939-8246-64c8d7cdbeec" (UID: "bc50a1c4-0675-4939-8246-64c8d7cdbeec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:00.727966 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.727935    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:00.727966 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.727961    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc50a1c4-0675-4939-8246-64c8d7cdbeec-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:00.728165 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.727972    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v29ms\" (UniqueName: \"kubernetes.io/projected/bc50a1c4-0675-4939-8246-64c8d7cdbeec-kube-api-access-v29ms\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:00.728165 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.727983    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc50a1c4-0675-4939-8246-64c8d7cdbeec-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:00.893056 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.893017    4837 generic.go:356] "Generic (PLEG): container finished" podID="b5463180-2162-4d32-b896-2ddd39e1b170" containerID="ade887165161bcd91e2d20d1fe660aff4cbb7a7d556932171ad3ab396aaefebe" exitCode=0
Nov 01 23:11:00.893239 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.893095    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" event={"ID":"b5463180-2162-4d32-b896-2ddd39e1b170","Type":"ContainerDied","Data":"ade887165161bcd91e2d20d1fe660aff4cbb7a7d556932171ad3ab396aaefebe"}
Nov 01 23:11:00.893239 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.893205    4837 scope.go:122] "RemoveContainer" containerID="6c9787cb3e6af8a184d49c392282fe7d8c10a539e2b1d61790467708e0c75d96"
Nov 01 23:11:00.895215 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.895002    4837 generic.go:356] "Generic (PLEG): container finished" podID="bc50a1c4-0675-4939-8246-64c8d7cdbeec" containerID="d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582" exitCode=0
Nov 01 23:11:00.895215 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.895077    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6" event={"ID":"bc50a1c4-0675-4939-8246-64c8d7cdbeec","Type":"ContainerDied","Data":"d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582"}
Nov 01 23:11:00.895215 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.895119    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6" event={"ID":"bc50a1c4-0675-4939-8246-64c8d7cdbeec","Type":"ContainerDied","Data":"b630f1e83baa4827f73e2dc40dbc79d2ce3d4c5cba1199bbeffd37b2c102e086"}
Nov 01 23:11:00.895215 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.895146    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"
Nov 01 23:11:00.905996 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.905957    4837 scope.go:122] "RemoveContainer" containerID="d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582"
Nov 01 23:11:00.916345 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.916326    4837 scope.go:122] "RemoveContainer" containerID="d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582"
Nov 01 23:11:00.916655 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:11:00.916631    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582\": container with ID starting with d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582 not found: ID does not exist" containerID="d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582"
Nov 01 23:11:00.916700 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:00.916672    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582"} err="failed to get container status \"d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582\": rpc error: code = NotFound desc = could not find container \"d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582\": container with ID starting with d89b962371c004a08fae51e3b2e6dc6b56178d0da30b0646de88b9911974e582 not found: ID does not exist"
Nov 01 23:11:01.021564 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.021541    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:11:01.051314 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.051291    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"]
Nov 01 23:11:01.055304 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.055279    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3eb631-887f9557d-78zj6"]
Nov 01 23:11:01.132650 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132599    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-socket\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.132766 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132684    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-istio-token\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-istio-token\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.132766 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132710    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-envoy\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.132766 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132729    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5463180-2162-4d32-b896-2ddd39e1b170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5463180-2162-4d32-b896-2ddd39e1b170-istio-podinfo\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.132766 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132753    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-kube-api-access-x88dx\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-kube-api-access-x88dx\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.132983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132847    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-socket" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:01.132983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132890    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-certs\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.132983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132917    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-credential-socket\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.132983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132944    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5463180-2162-4d32-b896-2ddd39e1b170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5463180-2162-4d32-b896-2ddd39e1b170-istiod-ca-cert\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.132983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.132979    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-data\") pod \"b5463180-2162-4d32-b896-2ddd39e1b170\" (UID: \"b5463180-2162-4d32-b896-2ddd39e1b170\") "
Nov 01 23:11:01.133292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.133173    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-certs" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:01.133373 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.133350    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-data" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:01.133516 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.133494    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5463180-2162-4d32-b896-2ddd39e1b170-istiod-ca-cert" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:01.133598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.133522    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.133598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.133541    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.133598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.133554    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.133598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.133557    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-credential-socket" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:01.134981 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.134954    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-istio-token" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:01.135115 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.135090    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-kube-api-access-x88dx" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "kube-api-access-x88dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:01.135324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.135307    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-envoy" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:01.135394 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.135377    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5463180-2162-4d32-b896-2ddd39e1b170-istio-podinfo" pod "b5463180-2162-4d32-b896-2ddd39e1b170" (UID: "b5463180-2162-4d32-b896-2ddd39e1b170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:11:01.234494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.234465    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.234494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.234489    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.234494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.234499    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5463180-2162-4d32-b896-2ddd39e1b170-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.234692 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.234508    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x88dx\" (UniqueName: \"kubernetes.io/projected/b5463180-2162-4d32-b896-2ddd39e1b170-kube-api-access-x88dx\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.234692 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.234519    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5463180-2162-4d32-b896-2ddd39e1b170-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.234692 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.234526    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5463180-2162-4d32-b896-2ddd39e1b170-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:01.903450 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.903409    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr" event={"ID":"b5463180-2162-4d32-b896-2ddd39e1b170","Type":"ContainerDied","Data":"b9148b91d9d60abdb1fa35ee158c77e7f70c90279f696bb8f89579b6932bfa70"}
Nov 01 23:11:01.903927 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.903477    4837 scope.go:122] "RemoveContainer" containerID="ade887165161bcd91e2d20d1fe660aff4cbb7a7d556932171ad3ab396aaefebe"
Nov 01 23:11:01.903927 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:01.903428    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"
Nov 01 23:11:02.105793 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:02.105757    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"]
Nov 01 23:11:02.152129 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:02.152070    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-675d1d-openshift-default-fcdb8bbdf-8gmxr"]
Nov 01 23:11:02.524599 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:02.524557    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" path="/var/lib/kubelet/pods/b5463180-2162-4d32-b896-2ddd39e1b170/volumes"
Nov 01 23:11:02.525150 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:02.525128    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc50a1c4-0675-4939-8246-64c8d7cdbeec" path="/var/lib/kubelet/pods/bc50a1c4-0675-4939-8246-64c8d7cdbeec/volumes"
Nov 01 23:11:03.861719 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:03.861688    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:11:12.687761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:12.687725    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"]
Nov 01 23:11:14.724333 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:14.724298    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"]
Nov 01 23:11:18.812315 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:11:18.812283    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:11:21.788293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:21.788256    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"]
Nov 01 23:11:26.789568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:26.789511    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m" podUID="7afb93f5-fcee-4434-bbfd-e297f8ea86cb" containerName="payload-processing" containerID="cri-o://ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c" gracePeriod=25
Nov 01 23:11:27.707494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:27.707458    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr" podUID="1236add6-9446-4069-b520-18789338dd10" containerName="main" containerID="cri-o://42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2" gracePeriod=45
Nov 01 23:11:27.981051 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:27.981024    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:11:28.032561 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.032537    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1236add6-9446-4069-b520-18789338dd10-tls-certs\" (UniqueName: \"kubernetes.io/secret/1236add6-9446-4069-b520-18789338dd10-tls-certs\") pod \"1236add6-9446-4069-b520-18789338dd10\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") "
Nov 01 23:11:28.032690 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.032569    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-tmp-dir\") pod \"1236add6-9446-4069-b520-18789338dd10\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") "
Nov 01 23:11:28.032690 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.032609    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-home\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-home\") pod \"1236add6-9446-4069-b520-18789338dd10\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") "
Nov 01 23:11:28.032690 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.032660    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-dshm\") pod \"1236add6-9446-4069-b520-18789338dd10\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") "
Nov 01 23:11:28.032878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.032708    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1236add6-9446-4069-b520-18789338dd10-kube-api-access-6d6js\" (UniqueName: \"kubernetes.io/projected/1236add6-9446-4069-b520-18789338dd10-kube-api-access-6d6js\") pod \"1236add6-9446-4069-b520-18789338dd10\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") "
Nov 01 23:11:28.032878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.032759    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-model-cache\") pod \"1236add6-9446-4069-b520-18789338dd10\" (UID: \"1236add6-9446-4069-b520-18789338dd10\") "
Nov 01 23:11:28.032878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.032794    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-tmp-dir" pod "1236add6-9446-4069-b520-18789338dd10" (UID: "1236add6-9446-4069-b520-18789338dd10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:28.033116 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.032934    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-home" pod "1236add6-9446-4069-b520-18789338dd10" (UID: "1236add6-9446-4069-b520-18789338dd10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:28.033194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.033122    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-model-cache" pod "1236add6-9446-4069-b520-18789338dd10" (UID: "1236add6-9446-4069-b520-18789338dd10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:28.033265 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.033248    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:28.033326 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.033271    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:28.033326 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.033282    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:28.034808 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.034784    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1236add6-9446-4069-b520-18789338dd10-tls-certs" pod "1236add6-9446-4069-b520-18789338dd10" (UID: "1236add6-9446-4069-b520-18789338dd10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:28.034808 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.034796    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-dshm" pod "1236add6-9446-4069-b520-18789338dd10" (UID: "1236add6-9446-4069-b520-18789338dd10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:28.034982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.034855    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1236add6-9446-4069-b520-18789338dd10-kube-api-access-6d6js" pod "1236add6-9446-4069-b520-18789338dd10" (UID: "1236add6-9446-4069-b520-18789338dd10"). InnerVolumeSpecName "kube-api-access-6d6js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:28.066615 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.066589    4837 generic.go:356] "Generic (PLEG): container finished" podID="1236add6-9446-4069-b520-18789338dd10" containerID="42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2" exitCode=0
Nov 01 23:11:28.066714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.066635    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"
Nov 01 23:11:28.066714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.066667    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr" event={"ID":"1236add6-9446-4069-b520-18789338dd10","Type":"ContainerDied","Data":"42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2"}
Nov 01 23:11:28.066714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.066711    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr" event={"ID":"1236add6-9446-4069-b520-18789338dd10","Type":"ContainerDied","Data":"f5da60e4f34bc81e833293cf86827fa42b07750df44390d55c3eeebe169f3c9b"}
Nov 01 23:11:28.066834 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.066731    4837 scope.go:122] "RemoveContainer" containerID="42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2"
Nov 01 23:11:28.076054 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.076029    4837 scope.go:122] "RemoveContainer" containerID="42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2"
Nov 01 23:11:28.076328 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:11:28.076309    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2\": container with ID starting with 42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2 not found: ID does not exist" containerID="42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2"
Nov 01 23:11:28.076378 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.076334    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2"} err="failed to get container status \"42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2\": rpc error: code = NotFound desc = could not find container \"42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2\": container with ID starting with 42b2707f0a78888d1b77a35023cc2c1c37d21c233d284ced0df2b3280a5b20d2 not found: ID does not exist"
Nov 01 23:11:28.134779 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.134758    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1236add6-9446-4069-b520-18789338dd10-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:28.134779 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.134780    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6js\" (UniqueName: \"kubernetes.io/projected/1236add6-9446-4069-b520-18789338dd10-kube-api-access-6d6js\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:28.134929 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.134790    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1236add6-9446-4069-b520-18789338dd10-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:28.208619 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.208592    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"]
Nov 01 23:11:28.212293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.212268    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-aa47d8/test-model-aa47d8-kserve-bb55cdfb5-dxhkr"]
Nov 01 23:11:28.523829 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:28.523801    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1236add6-9446-4069-b520-18789338dd10" path="/var/lib/kubelet/pods/1236add6-9446-4069-b520-18789338dd10/volumes"
Nov 01 23:11:29.744703 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:29.744667    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main" containerID="cri-o://e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495" gracePeriod=45
Nov 01 23:11:30.006816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.006762    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 23:11:30.052065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052018    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-dshm\") pod \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") "
Nov 01 23:11:30.052183 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052077    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-model-cache\") pod \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") "
Nov 01 23:11:30.052183 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052123    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tmp-dir\") pod \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") "
Nov 01 23:11:30.052183 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052167    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tls-certs\") pod \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") "
Nov 01 23:11:30.052292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052247    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbe9044-faad-4c63-bc0d-292c2e910a2d-kube-api-access-hjghj\" (UniqueName: \"kubernetes.io/projected/0fbe9044-faad-4c63-bc0d-292c2e910a2d-kube-api-access-hjghj\") pod \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") "
Nov 01 23:11:30.052292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052281    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-home\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-home\") pod \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\" (UID: \"0fbe9044-faad-4c63-bc0d-292c2e910a2d\") "
Nov 01 23:11:30.052420 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052394    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-model-cache" pod "0fbe9044-faad-4c63-bc0d-292c2e910a2d" (UID: "0fbe9044-faad-4c63-bc0d-292c2e910a2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.052487 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052407    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tmp-dir" pod "0fbe9044-faad-4c63-bc0d-292c2e910a2d" (UID: "0fbe9044-faad-4c63-bc0d-292c2e910a2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.052627 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052599    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-home" pod "0fbe9044-faad-4c63-bc0d-292c2e910a2d" (UID: "0fbe9044-faad-4c63-bc0d-292c2e910a2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.052750 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052689    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.052750 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052702    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.052750 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.052711    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.054350 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.054324    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-dshm" pod "0fbe9044-faad-4c63-bc0d-292c2e910a2d" (UID: "0fbe9044-faad-4c63-bc0d-292c2e910a2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.054444 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.054426    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbe9044-faad-4c63-bc0d-292c2e910a2d-kube-api-access-hjghj" pod "0fbe9044-faad-4c63-bc0d-292c2e910a2d" (UID: "0fbe9044-faad-4c63-bc0d-292c2e910a2d"). InnerVolumeSpecName "kube-api-access-hjghj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:30.054490 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.054478    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tls-certs" pod "0fbe9044-faad-4c63-bc0d-292c2e910a2d" (UID: "0fbe9044-faad-4c63-bc0d-292c2e910a2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:30.079509 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.079486    4837 generic.go:356] "Generic (PLEG): container finished" podID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerID="e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495" exitCode=0
Nov 01 23:11:30.079613 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.079533    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"
Nov 01 23:11:30.079613 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.079541    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" event={"ID":"0fbe9044-faad-4c63-bc0d-292c2e910a2d","Type":"ContainerDied","Data":"e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495"}
Nov 01 23:11:30.079613 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.079581    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9" event={"ID":"0fbe9044-faad-4c63-bc0d-292c2e910a2d","Type":"ContainerDied","Data":"4d2123779aa0536fb5d9016609dd32fbb5e67a88f9e3276b653ed95ebc2d346d"}
Nov 01 23:11:30.079613 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.079598    4837 scope.go:122] "RemoveContainer" containerID="e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495"
Nov 01 23:11:30.088890 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.088867    4837 scope.go:122] "RemoveContainer" containerID="e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495"
Nov 01 23:11:30.089179 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:11:30.089157    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495\": container with ID starting with e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495 not found: ID does not exist" containerID="e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495"
Nov 01 23:11:30.089251 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.089186    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495"} err="failed to get container status \"e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495\": rpc error: code = NotFound desc = could not find container \"e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495\": container with ID starting with e79f1462f234fc72d3025e8af75b5ce19fec549af22021ee0799e59cfb5cc495 not found: ID does not exist"
Nov 01 23:11:30.153976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.153950    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbe9044-faad-4c63-bc0d-292c2e910a2d-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.153976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.153972    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbe9044-faad-4c63-bc0d-292c2e910a2d-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.153976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.153983    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjghj\" (UniqueName: \"kubernetes.io/projected/0fbe9044-faad-4c63-bc0d-292c2e910a2d-kube-api-access-hjghj\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.223678 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.223642    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"]
Nov 01 23:11:30.227098 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.227073    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/premium-w6-a6dd2f-kserve-646d496bd9-8l5w9"]
Nov 01 23:11:30.524330 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:30.524298    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" path="/var/lib/kubelet/pods/0fbe9044-faad-4c63-bc0d-292c2e910a2d/volumes"
Nov 01 23:11:34.574824 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:34.574785    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m" podUID="7afb93f5-fcee-4434-bbfd-e297f8ea86cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:11:37.164855 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:37.164821    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:11:44.575629 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:44.575595    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m" podUID="7afb93f5-fcee-4434-bbfd-e297f8ea86cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:11:47.055086 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.055014    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 23:11:47.119900 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.119872    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-plugins-config-volume\") pod \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\" (UID: \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\") "
Nov 01 23:11:47.120066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.119930    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-kube-api-access-mgqks\" (UniqueName: \"kubernetes.io/projected/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-kube-api-access-mgqks\") pod \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\" (UID: \"7afb93f5-fcee-4434-bbfd-e297f8ea86cb\") "
Nov 01 23:11:47.120309 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.120286    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-plugins-config-volume" pod "7afb93f5-fcee-4434-bbfd-e297f8ea86cb" (UID: "7afb93f5-fcee-4434-bbfd-e297f8ea86cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:47.121992 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.121964    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-kube-api-access-mgqks" pod "7afb93f5-fcee-4434-bbfd-e297f8ea86cb" (UID: "7afb93f5-fcee-4434-bbfd-e297f8ea86cb"). InnerVolumeSpecName "kube-api-access-mgqks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:47.171829 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.171797    4837 generic.go:356] "Generic (PLEG): container finished" podID="7afb93f5-fcee-4434-bbfd-e297f8ea86cb" containerID="ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c" exitCode=0
Nov 01 23:11:47.171975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.171848    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"
Nov 01 23:11:47.171975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.171887    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m" event={"ID":"7afb93f5-fcee-4434-bbfd-e297f8ea86cb","Type":"ContainerDied","Data":"ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c"}
Nov 01 23:11:47.171975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.171942    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m" event={"ID":"7afb93f5-fcee-4434-bbfd-e297f8ea86cb","Type":"ContainerDied","Data":"c0007e381682628a86f10e0ebf1fd103b9b74b238ced4e5b633c04eab90d68ce"}
Nov 01 23:11:47.171975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.171970    4837 scope.go:122] "RemoveContainer" containerID="ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c"
Nov 01 23:11:47.183499 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.183479    4837 scope.go:122] "RemoveContainer" containerID="ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c"
Nov 01 23:11:47.183807 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:11:47.183774    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c\": container with ID starting with ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c not found: ID does not exist" containerID="ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c"
Nov 01 23:11:47.183888 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.183812    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c"} err="failed to get container status \"ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c\": rpc error: code = NotFound desc = could not find container \"ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c\": container with ID starting with ebc07ffcf09466bbef3330f7dd034ab8522edca518b0a79cb9e112661347ad7c not found: ID does not exist"
Nov 01 23:11:47.220953 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.220923    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:47.220953 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.220945    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgqks\" (UniqueName: \"kubernetes.io/projected/7afb93f5-fcee-4434-bbfd-e297f8ea86cb-kube-api-access-mgqks\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:47.326568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.326502    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"]
Nov 01 23:11:47.331850 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:47.331823    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a6dd2f-54b67695cc-dfx8m"]
Nov 01 23:11:48.534256 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:48.534224    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7afb93f5-fcee-4434-bbfd-e297f8ea86cb" path="/var/lib/kubelet/pods/7afb93f5-fcee-4434-bbfd-e297f8ea86cb/volumes"
Nov 01 23:11:48.812542 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:11:48.812466    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:11:54.897544 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:54.897511    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"]
Nov 01 23:11:54.898141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:11:54.897706    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy" containerID="cri-o://edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df" gracePeriod=30
Nov 01 23:12:00.150465 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.150434    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:12:00.243266 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.243235    4837 generic.go:356] "Generic (PLEG): container finished" podID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerID="edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df" exitCode=0
Nov 01 23:12:00.243411 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.243281    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"
Nov 01 23:12:00.243411 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.243304    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" event={"ID":"0165cb48-bf04-4815-a668-31a3af4f64ed","Type":"ContainerDied","Data":"edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df"}
Nov 01 23:12:00.243411 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.243357    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b" event={"ID":"0165cb48-bf04-4815-a668-31a3af4f64ed","Type":"ContainerDied","Data":"beadd174fa17c3e9efdd8672ba9690e658a941d08e8943c060b9700a31131bfb"}
Nov 01 23:12:00.243411 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.243386    4837 scope.go:122] "RemoveContainer" containerID="edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df"
Nov 01 23:12:00.254910 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.254877    4837 scope.go:122] "RemoveContainer" containerID="c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6"
Nov 01 23:12:00.257460 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257441    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-token\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-token\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.257572 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257473    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-podinfo\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.257572 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257501    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-socket\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.257572 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257535    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0165cb48-bf04-4815-a668-31a3af4f64ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165cb48-bf04-4815-a668-31a3af4f64ed-istiod-ca-cert\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.257572 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257559    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-credential-socket\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.257803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257591    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-certs\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.257803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257635    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-data\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.257803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257661    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-kube-api-access-qcbc8\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-kube-api-access-qcbc8\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.257803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257734    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-envoy\") pod \"0165cb48-bf04-4815-a668-31a3af4f64ed\" (UID: \"0165cb48-bf04-4815-a668-31a3af4f64ed\") "
Nov 01 23:12:00.258171 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.257872    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-credential-socket" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:00.258245 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.258172    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0165cb48-bf04-4815-a668-31a3af4f64ed-istiod-ca-cert" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:00.258245 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.258178    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-certs" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:00.258444 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.258398    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0165cb48-bf04-4815-a668-31a3af4f64ed-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.258444 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.258423    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.258444 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.258425    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-data" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:00.258444 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.258438    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.258759 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.258495    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-socket" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:00.260023 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.259997    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-token" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:00.260023 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.260009    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-kube-api-access-qcbc8" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "kube-api-access-qcbc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:00.260172 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.260150    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-envoy" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:00.260267 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.260252    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-podinfo" pod "0165cb48-bf04-4815-a668-31a3af4f64ed" (UID: "0165cb48-bf04-4815-a668-31a3af4f64ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:12:00.280148 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.280131    4837 scope.go:122] "RemoveContainer" containerID="edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df"
Nov 01 23:12:00.280393 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:00.280374    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df\": container with ID starting with edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df not found: ID does not exist" containerID="edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df"
Nov 01 23:12:00.280476 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.280407    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df"} err="failed to get container status \"edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df\": rpc error: code = NotFound desc = could not find container \"edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df\": container with ID starting with edb93a64cb38fb3266e8f4db8f248a2939be9d5f303ce8b4f2233fbbfcf969df not found: ID does not exist"
Nov 01 23:12:00.280476 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.280429    4837 scope.go:122] "RemoveContainer" containerID="c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6"
Nov 01 23:12:00.280677 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:00.280659    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6\": container with ID starting with c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6 not found: ID does not exist" containerID="c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6"
Nov 01 23:12:00.280723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.280682    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6"} err="failed to get container status \"c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6\": rpc error: code = NotFound desc = could not find container \"c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6\": container with ID starting with c38e3330a27d6ffcbaaaf54947f52f1a124e61acc3b0f35327abbdf9a94131c6 not found: ID does not exist"
Nov 01 23:12:00.359244 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.359217    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.359244 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.359240    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.359244 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.359251    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.359447 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.359260    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.359447 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.359269    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcbc8\" (UniqueName: \"kubernetes.io/projected/0165cb48-bf04-4815-a668-31a3af4f64ed-kube-api-access-qcbc8\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.359447 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.359278    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0165cb48-bf04-4815-a668-31a3af4f64ed-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.535864 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.535835    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"]
Nov 01 23:12:00.747340 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.747308    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"]
Nov 01 23:12:00.756816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:00.756769    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4af44d-openshift-default-58849ddc6c-ggv8b"]
Nov 01 23:12:02.523925 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:02.523886    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" path="/var/lib/kubelet/pods/0165cb48-bf04-4815-a668-31a3af4f64ed/volumes"
Nov 01 23:12:02.916457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:02.916434    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:12:05.537063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:05.537002    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk" podUID="0ab1b3f3-5281-4815-87cd-df2dbea8b90c" containerName="payload-processing" containerID="cri-o://f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0" gracePeriod=25
Nov 01 23:12:12.972918 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:12.972883    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk" podUID="0ab1b3f3-5281-4815-87cd-df2dbea8b90c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:12:18.812123 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:18.812022    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:12:22.973270 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:22.973236    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk" podUID="0ab1b3f3-5281-4815-87cd-df2dbea8b90c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:12:25.798988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:25.798918    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:12:25.939018 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:25.938988    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-kube-api-access-55z7n\" (UniqueName: \"kubernetes.io/projected/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-kube-api-access-55z7n\") pod \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\" (UID: \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\") "
Nov 01 23:12:25.939225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:25.939106    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-plugins-config-volume\") pod \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\" (UID: \"0ab1b3f3-5281-4815-87cd-df2dbea8b90c\") "
Nov 01 23:12:25.939480 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:25.939455    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-plugins-config-volume" pod "0ab1b3f3-5281-4815-87cd-df2dbea8b90c" (UID: "0ab1b3f3-5281-4815-87cd-df2dbea8b90c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:25.941217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:25.941193    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-kube-api-access-55z7n" pod "0ab1b3f3-5281-4815-87cd-df2dbea8b90c" (UID: "0ab1b3f3-5281-4815-87cd-df2dbea8b90c"). InnerVolumeSpecName "kube-api-access-55z7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:26.040672 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.040643    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55z7n\" (UniqueName: \"kubernetes.io/projected/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-kube-api-access-55z7n\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:26.040672 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.040670    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab1b3f3-5281-4815-87cd-df2dbea8b90c-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:26.384871 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.384837    4837 generic.go:356] "Generic (PLEG): container finished" podID="0ab1b3f3-5281-4815-87cd-df2dbea8b90c" containerID="f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0" exitCode=0
Nov 01 23:12:26.385082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.384888    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"
Nov 01 23:12:26.385082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.384920    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk" event={"ID":"0ab1b3f3-5281-4815-87cd-df2dbea8b90c","Type":"ContainerDied","Data":"f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0"}
Nov 01 23:12:26.385082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.384974    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk" event={"ID":"0ab1b3f3-5281-4815-87cd-df2dbea8b90c","Type":"ContainerDied","Data":"3d51709dce3c0934f8e1f0d01ab94c1e535bd6424b4accdac218fed95c05305f"}
Nov 01 23:12:26.385082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.384997    4837 scope.go:122] "RemoveContainer" containerID="f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0"
Nov 01 23:12:26.394514 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.394497    4837 scope.go:122] "RemoveContainer" containerID="f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0"
Nov 01 23:12:26.394782 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:26.394765    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0\": container with ID starting with f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0 not found: ID does not exist" containerID="f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0"
Nov 01 23:12:26.394832 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.394792    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0"} err="failed to get container status \"f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0\": rpc error: code = NotFound desc = could not find container \"f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0\": container with ID starting with f10f3b5adcabfb39754d61eb38feacdd1b80f3d09432c5b657e865bec41fa2f0 not found: ID does not exist"
Nov 01 23:12:26.539409 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.539378    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"]
Nov 01 23:12:26.542769 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:26.542742    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa47d8-75b4996bf5-ww9hk"]
Nov 01 23:12:27.163911 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.163879    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"]
Nov 01 23:12:27.164363 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.164085    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j" podUID="e17dc971-5864-4d0a-807e-78e593e3fec1" containerName="maas-api" containerID="cri-o://3afc0e51525957d3c0619ff56de993bfb562e55081d17c09f0a8babe87ba135f" gracePeriod=30
Nov 01 23:12:27.407005 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.406974    4837 generic.go:356] "Generic (PLEG): container finished" podID="e17dc971-5864-4d0a-807e-78e593e3fec1" containerID="3afc0e51525957d3c0619ff56de993bfb562e55081d17c09f0a8babe87ba135f" exitCode=0
Nov 01 23:12:27.407164 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.407029    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j" event={"ID":"e17dc971-5864-4d0a-807e-78e593e3fec1","Type":"ContainerDied","Data":"3afc0e51525957d3c0619ff56de993bfb562e55081d17c09f0a8babe87ba135f"}
Nov 01 23:12:27.428412 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.428386    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:12:27.576471 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.576429    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls\") pod \"e17dc971-5864-4d0a-807e-78e593e3fec1\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") "
Nov 01 23:12:27.576990 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.576962    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-metrics-certs\") pod \"e17dc971-5864-4d0a-807e-78e593e3fec1\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") "
Nov 01 23:12:27.577160 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.577031    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17dc971-5864-4d0a-807e-78e593e3fec1-kube-api-access-rhk9n\" (UniqueName: \"kubernetes.io/projected/e17dc971-5864-4d0a-807e-78e593e3fec1-kube-api-access-rhk9n\") pod \"e17dc971-5864-4d0a-807e-78e593e3fec1\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") "
Nov 01 23:12:27.577241 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.577188    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17dc971-5864-4d0a-807e-78e593e3fec1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17dc971-5864-4d0a-807e-78e593e3fec1-openshift-service-ca\") pod \"e17dc971-5864-4d0a-807e-78e593e3fec1\" (UID: \"e17dc971-5864-4d0a-807e-78e593e3fec1\") "
Nov 01 23:12:27.577943 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.577911    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17dc971-5864-4d0a-807e-78e593e3fec1-openshift-service-ca" pod "e17dc971-5864-4d0a-807e-78e593e3fec1" (UID: "e17dc971-5864-4d0a-807e-78e593e3fec1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:27.578351 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.578325    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17dc971-5864-4d0a-807e-78e593e3fec1-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:27.581321 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.581294    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17dc971-5864-4d0a-807e-78e593e3fec1-kube-api-access-rhk9n" pod "e17dc971-5864-4d0a-807e-78e593e3fec1" (UID: "e17dc971-5864-4d0a-807e-78e593e3fec1"). InnerVolumeSpecName "kube-api-access-rhk9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:27.581858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.581833    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls" pod "e17dc971-5864-4d0a-807e-78e593e3fec1" (UID: "e17dc971-5864-4d0a-807e-78e593e3fec1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:27.582584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.582566    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-metrics-certs" pod "e17dc971-5864-4d0a-807e-78e593e3fec1" (UID: "e17dc971-5864-4d0a-807e-78e593e3fec1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:27.682777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.681932    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:27.682777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.681967    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17dc971-5864-4d0a-807e-78e593e3fec1-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:27.682777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:27.681983    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhk9n\" (UniqueName: \"kubernetes.io/projected/e17dc971-5864-4d0a-807e-78e593e3fec1-kube-api-access-rhk9n\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:28.413891 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:28.413860    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"
Nov 01 23:12:28.413891 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:28.413881    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j" event={"ID":"e17dc971-5864-4d0a-807e-78e593e3fec1","Type":"ContainerDied","Data":"4099c41bdf8e5c66d62d83a9a66693ac5f44c165234d74d8d5d8deb4c1f523de"}
Nov 01 23:12:28.414379 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:28.413964    4837 scope.go:122] "RemoveContainer" containerID="3afc0e51525957d3c0619ff56de993bfb562e55081d17c09f0a8babe87ba135f"
Nov 01 23:12:28.523552 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:28.523525    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab1b3f3-5281-4815-87cd-df2dbea8b90c" path="/var/lib/kubelet/pods/0ab1b3f3-5281-4815-87cd-df2dbea8b90c/volumes"
Nov 01 23:12:28.572961 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:28.572936    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"]
Nov 01 23:12:28.619104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:28.619078    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa47d8-6c94dffbc5-bdz6j"]
Nov 01 23:12:30.524007 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:30.523972    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17dc971-5864-4d0a-807e-78e593e3fec1" path="/var/lib/kubelet/pods/e17dc971-5864-4d0a-807e-78e593e3fec1/volumes"
Nov 01 23:12:34.057494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:34.057462    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:12:36.526860 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:36.526828    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"]
Nov 01 23:12:36.527315 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:36.527059    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy" containerID="cri-o://d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f" gracePeriod=30
Nov 01 23:12:41.776303 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.776279    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:12:41.857060 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.856985    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-podinfo\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857060 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857017    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istiod-ca-cert\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857229 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857088    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-envoy\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857229 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857122    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-token\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857229 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857180    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-credential-socket\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857229 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857208    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-data\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857321    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-socket\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857356    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-certs\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857409    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-kube-api-access-svlfl\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-kube-api-access-svlfl\") pod \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\" (UID: \"2b2f5e83-f48d-422e-a12c-65942e5ceb5f\") "
Nov 01 23:12:41.857634 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857612    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-credential-socket" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:41.857705 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857634    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-data" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:41.857705 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857651    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istiod-ca-cert" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:41.857824 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857652    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-certs" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:41.857904 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857882    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-socket" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:41.857975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857899    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.857975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857922    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.857975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857937    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.857975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.857951    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.859548 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.859520    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-token" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:41.859548 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.859509    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-podinfo" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:12:41.859685 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.859559    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-envoy" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:41.859685 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.859630    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-kube-api-access-svlfl" pod "2b2f5e83-f48d-422e-a12c-65942e5ceb5f" (UID: "2b2f5e83-f48d-422e-a12c-65942e5ceb5f"). InnerVolumeSpecName "kube-api-access-svlfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:41.959441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.959410    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.959441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.959439    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svlfl\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-kube-api-access-svlfl\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.959636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.959453    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.959636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.959466    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.959636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:41.959477    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b2f5e83-f48d-422e-a12c-65942e5ceb5f-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:42.494112 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.494074    4837 generic.go:356] "Generic (PLEG): container finished" podID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerID="d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f" exitCode=0
Nov 01 23:12:42.494292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.494146    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"
Nov 01 23:12:42.494292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.494169    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" event={"ID":"2b2f5e83-f48d-422e-a12c-65942e5ceb5f","Type":"ContainerDied","Data":"d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f"}
Nov 01 23:12:42.494292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.494219    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb" event={"ID":"2b2f5e83-f48d-422e-a12c-65942e5ceb5f","Type":"ContainerDied","Data":"d16c55bae2b1ca0d951874225b75d146fde02a8d7cb9020459e67237befae01b"}
Nov 01 23:12:42.494292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.494242    4837 scope.go:122] "RemoveContainer" containerID="d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f"
Nov 01 23:12:42.512563 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.512540    4837 scope.go:122] "RemoveContainer" containerID="480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df"
Nov 01 23:12:42.521380 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.521356    4837 scope.go:122] "RemoveContainer" containerID="d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f"
Nov 01 23:12:42.521691 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:42.521670    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f\": container with ID starting with d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f not found: ID does not exist" containerID="d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f"
Nov 01 23:12:42.521763 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.521702    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f"} err="failed to get container status \"d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f\": rpc error: code = NotFound desc = could not find container \"d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f\": container with ID starting with d8b3dc7d9f2ae5a4d5ef8cfb52433c23bff53c0f024b7494f9a7c90e97a5a53f not found: ID does not exist"
Nov 01 23:12:42.521763 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.521724    4837 scope.go:122] "RemoveContainer" containerID="480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df"
Nov 01 23:12:42.521958 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:42.521942    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df\": container with ID starting with 480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df not found: ID does not exist" containerID="480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df"
Nov 01 23:12:42.522017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.521966    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df"} err="failed to get container status \"480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df\": rpc error: code = NotFound desc = could not find container \"480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df\": container with ID starting with 480da6b1e9208fa112d9b8ddc5c43d5d9e130802dfadd145c06ea3c93a3b52df not found: ID does not exist"
Nov 01 23:12:42.694914 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.694885    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"]
Nov 01 23:12:42.741054 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:42.741020    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-aa47d8-openshift-default-679846449c-7vlxb"]
Nov 01 23:12:43.508464 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.508436    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"]
Nov 01 23:12:43.508920 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.508611    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc" podUID="fb963b29-0c1b-439c-a887-763eff140b30" containerName="maas-api" containerID="cri-o://1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e" gracePeriod=30
Nov 01 23:12:43.770189 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.770134    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 23:12:43.881097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.881065    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb963b29-0c1b-439c-a887-763eff140b30-kube-api-access-7ql9q\" (UniqueName: \"kubernetes.io/projected/fb963b29-0c1b-439c-a887-763eff140b30-kube-api-access-7ql9q\") pod \"fb963b29-0c1b-439c-a887-763eff140b30\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") "
Nov 01 23:12:43.881097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.881159    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-metrics-certs\") pod \"fb963b29-0c1b-439c-a887-763eff140b30\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") "
Nov 01 23:12:43.881097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.881227    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb963b29-0c1b-439c-a887-763eff140b30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb963b29-0c1b-439c-a887-763eff140b30-openshift-service-ca\") pod \"fb963b29-0c1b-439c-a887-763eff140b30\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") "
Nov 01 23:12:43.881097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.881274    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-maas-api-tls\") pod \"fb963b29-0c1b-439c-a887-763eff140b30\" (UID: \"fb963b29-0c1b-439c-a887-763eff140b30\") "
Nov 01 23:12:43.881952 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.881925    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb963b29-0c1b-439c-a887-763eff140b30-openshift-service-ca" pod "fb963b29-0c1b-439c-a887-763eff140b30" (UID: "fb963b29-0c1b-439c-a887-763eff140b30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:43.884013 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.883987    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-metrics-certs" pod "fb963b29-0c1b-439c-a887-763eff140b30" (UID: "fb963b29-0c1b-439c-a887-763eff140b30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:43.884269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.884236    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb963b29-0c1b-439c-a887-763eff140b30-kube-api-access-7ql9q" pod "fb963b29-0c1b-439c-a887-763eff140b30" (UID: "fb963b29-0c1b-439c-a887-763eff140b30"). InnerVolumeSpecName "kube-api-access-7ql9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:43.884363 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.884279    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-maas-api-tls" pod "fb963b29-0c1b-439c-a887-763eff140b30" (UID: "fb963b29-0c1b-439c-a887-763eff140b30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:43.982831 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.982796    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb963b29-0c1b-439c-a887-763eff140b30-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:43.982831 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.982830    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:43.983028 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.982845    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ql9q\" (UniqueName: \"kubernetes.io/projected/fb963b29-0c1b-439c-a887-763eff140b30-kube-api-access-7ql9q\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:43.983028 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:43.982862    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb963b29-0c1b-439c-a887-763eff140b30-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.507117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.507082    4837 generic.go:356] "Generic (PLEG): container finished" podID="fb963b29-0c1b-439c-a887-763eff140b30" containerID="1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e" exitCode=0
Nov 01 23:12:44.507298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.507139    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"
Nov 01 23:12:44.507298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.507142    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc" event={"ID":"fb963b29-0c1b-439c-a887-763eff140b30","Type":"ContainerDied","Data":"1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e"}
Nov 01 23:12:44.507298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.507271    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc" event={"ID":"fb963b29-0c1b-439c-a887-763eff140b30","Type":"ContainerDied","Data":"80367921bb488ca86884b11e4b84a8e6a2237f7bed7965c2baf538bf7600da21"}
Nov 01 23:12:44.507298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.507296    4837 scope.go:122] "RemoveContainer" containerID="1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e"
Nov 01 23:12:44.517164 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.516920    4837 scope.go:122] "RemoveContainer" containerID="1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e"
Nov 01 23:12:44.517404 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:44.517225    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e\": container with ID starting with 1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e not found: ID does not exist" containerID="1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e"
Nov 01 23:12:44.517404 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.517257    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e"} err="failed to get container status \"1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e\": rpc error: code = NotFound desc = could not find container \"1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e\": container with ID starting with 1bc851ae445d5b122c1308630a895e43b8fe79aed6805f3d3db8dc6f158a919e not found: ID does not exist"
Nov 01 23:12:44.524224 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.524205    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" path="/var/lib/kubelet/pods/2b2f5e83-f48d-422e-a12c-65942e5ceb5f/volumes"
Nov 01 23:12:44.660005 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.659973    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"]
Nov 01 23:12:44.705888 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:44.705858    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6dd2f-5786fbbf64-wd2xc"]
Nov 01 23:12:46.523934 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:46.523905    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb963b29-0c1b-439c-a887-763eff140b30" path="/var/lib/kubelet/pods/fb963b29-0c1b-439c-a887-763eff140b30/volumes"
Nov 01 23:12:48.811535 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:48.811496    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:12:50.372169 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:50.372129    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:12:53.883318 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:53.883283    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"]
Nov 01 23:12:53.883737 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:53.883535    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy" containerID="cri-o://fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b" gracePeriod=30
Nov 01 23:12:59.137873 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.137848    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 23:12:59.154956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.154890    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istiod-ca-cert\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.154956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.154933    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-podinfo\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.155168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.154966    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-credential-socket\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.155168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.154999    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-certs\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.155168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155031    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-socket\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.155168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155080    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-token\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-token\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.155168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155124    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-envoy\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.155430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155189    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-data\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.155430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155283    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-kube-api-access-nqb8l\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-kube-api-access-nqb8l\") pod \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\" (UID: \"2baff4f5-c9e3-4860-9a29-2b36a1e09547\") "
Nov 01 23:12:59.155430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155352    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istiod-ca-cert" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:59.155430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155326    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-certs" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:59.155430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155370    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-credential-socket" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:59.155430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155385    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-socket" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:59.155713 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155545    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-data" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:59.155713 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155692    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.155713 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155711    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.155883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155726    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.155883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155742    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.155883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.155755    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.157262 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.157231    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-podinfo" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:12:59.157375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.157340    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-token" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:59.157435 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.157405    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-kube-api-access-nqb8l" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "kube-api-access-nqb8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:59.157477 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.157432    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-envoy" pod "2baff4f5-c9e3-4860-9a29-2b36a1e09547" (UID: "2baff4f5-c9e3-4860-9a29-2b36a1e09547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:59.256440 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.256413    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.256440 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.256436    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.256585 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.256444    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2baff4f5-c9e3-4860-9a29-2b36a1e09547-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.256585 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.256453    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqb8l\" (UniqueName: \"kubernetes.io/projected/2baff4f5-c9e3-4860-9a29-2b36a1e09547-kube-api-access-nqb8l\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:59.580656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.580623    4837 generic.go:356] "Generic (PLEG): container finished" podID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerID="fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b" exitCode=0
Nov 01 23:12:59.580836 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.580670    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"
Nov 01 23:12:59.580836 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.580704    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" event={"ID":"2baff4f5-c9e3-4860-9a29-2b36a1e09547","Type":"ContainerDied","Data":"fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b"}
Nov 01 23:12:59.580836 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.580760    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj" event={"ID":"2baff4f5-c9e3-4860-9a29-2b36a1e09547","Type":"ContainerDied","Data":"1c7b38654260c23a54f8e01282d5fbba2e40f63b6f01dbb96970d14218f8aad6"}
Nov 01 23:12:59.580836 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.580789    4837 scope.go:122] "RemoveContainer" containerID="fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b"
Nov 01 23:12:59.590770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.590755    4837 scope.go:122] "RemoveContainer" containerID="f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363"
Nov 01 23:12:59.600721 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.600688    4837 scope.go:122] "RemoveContainer" containerID="fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b"
Nov 01 23:12:59.600995 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:59.600978    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b\": container with ID starting with fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b not found: ID does not exist" containerID="fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b"
Nov 01 23:12:59.601106 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.601006    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b"} err="failed to get container status \"fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b\": rpc error: code = NotFound desc = could not find container \"fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b\": container with ID starting with fd7a0996003152f82b6acf5ec4c0c785483192e2f7f9bd172954372e26c2cb7b not found: ID does not exist"
Nov 01 23:12:59.601106 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.601029    4837 scope.go:122] "RemoveContainer" containerID="f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363"
Nov 01 23:12:59.601333 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:12:59.601316    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363\": container with ID starting with f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363 not found: ID does not exist" containerID="f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363"
Nov 01 23:12:59.601387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.601339    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363"} err="failed to get container status \"f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363\": rpc error: code = NotFound desc = could not find container \"f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363\": container with ID starting with f1b3f3a8785465c5bf48dd0e0d086c49cecc4abad3b424818208a0eedf775363 not found: ID does not exist"
Nov 01 23:12:59.750638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.750609    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"]
Nov 01 23:12:59.760851 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:12:59.760824    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-646881-openshift-default-76f4f7bd65-svcbj"]
Nov 01 23:13:00.523474 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:00.523444    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" path="/var/lib/kubelet/pods/2baff4f5-c9e3-4860-9a29-2b36a1e09547/volumes"
Nov 01 23:13:06.229653 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:06.229622    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"]
Nov 01 23:13:11.232748 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.232708    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t" podUID="4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14" containerName="payload-processing" containerID="cri-o://b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210" gracePeriod=25
Nov 01 23:13:11.491956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.491904    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 23:13:11.570538 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.570512    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-kube-api-access-vn2sw\" (UniqueName: \"kubernetes.io/projected/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-kube-api-access-vn2sw\") pod \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\" (UID: \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\") "
Nov 01 23:13:11.570700 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.570575    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-plugins-config-volume\") pod \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\" (UID: \"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14\") "
Nov 01 23:13:11.570956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.570932    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-plugins-config-volume" pod "4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14" (UID: "4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:11.572635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.572613    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-kube-api-access-vn2sw" pod "4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14" (UID: "4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14"). InnerVolumeSpecName "kube-api-access-vn2sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:11.639234 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.639199    4837 generic.go:356] "Generic (PLEG): container finished" podID="4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14" containerID="b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210" exitCode=0
Nov 01 23:13:11.639394 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.639245    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"
Nov 01 23:13:11.639394 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.639273    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t" event={"ID":"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14","Type":"ContainerDied","Data":"b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210"}
Nov 01 23:13:11.639394 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.639329    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t" event={"ID":"4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14","Type":"ContainerDied","Data":"d44500fadf239d8c095ec2a3e2c013f1b37eb0364ed64bc093c2e6cc57bc591f"}
Nov 01 23:13:11.639394 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.639354    4837 scope.go:122] "RemoveContainer" containerID="b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210"
Nov 01 23:13:11.648670 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.648653    4837 scope.go:122] "RemoveContainer" containerID="b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210"
Nov 01 23:13:11.648936 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:11.648918    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210\": container with ID starting with b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210 not found: ID does not exist" containerID="b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210"
Nov 01 23:13:11.648988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.648944    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210"} err="failed to get container status \"b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210\": rpc error: code = NotFound desc = could not find container \"b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210\": container with ID starting with b234a08e1adf86b645ce0b4f23d7e8e3cfb5123351083b9d28bb57a5fe35a210 not found: ID does not exist"
Nov 01 23:13:11.671597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.671572    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn2sw\" (UniqueName: \"kubernetes.io/projected/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-kube-api-access-vn2sw\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:11.671597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.671595    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:11.791293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.791217    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"]
Nov 01 23:13:11.794956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:11.794926    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8cc47a-7b4c478867-nqn7t"]
Nov 01 23:13:12.523907 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:12.523877    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14" path="/var/lib/kubelet/pods/4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14/volumes"
Nov 01 23:13:17.097097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.097069    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"]
Nov 01 23:13:17.097516 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.097276    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d" podUID="28ebabbb-df4d-496d-95bf-0a6e07b92520" containerName="maas-api" containerID="cri-o://480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092" gracePeriod=30
Nov 01 23:13:17.360933 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.360879    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 23:13:17.429699 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.429675    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ebabbb-df4d-496d-95bf-0a6e07b92520-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ebabbb-df4d-496d-95bf-0a6e07b92520-openshift-service-ca\") pod \"28ebabbb-df4d-496d-95bf-0a6e07b92520\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") "
Nov 01 23:13:17.429835 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.429711    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls\") pod \"28ebabbb-df4d-496d-95bf-0a6e07b92520\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") "
Nov 01 23:13:17.429835 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.429776    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-metrics-certs\") pod \"28ebabbb-df4d-496d-95bf-0a6e07b92520\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") "
Nov 01 23:13:17.429922 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.429884    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ebabbb-df4d-496d-95bf-0a6e07b92520-kube-api-access-bxhsr\" (UniqueName: \"kubernetes.io/projected/28ebabbb-df4d-496d-95bf-0a6e07b92520-kube-api-access-bxhsr\") pod \"28ebabbb-df4d-496d-95bf-0a6e07b92520\" (UID: \"28ebabbb-df4d-496d-95bf-0a6e07b92520\") "
Nov 01 23:13:17.430141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.430118    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ebabbb-df4d-496d-95bf-0a6e07b92520-openshift-service-ca" pod "28ebabbb-df4d-496d-95bf-0a6e07b92520" (UID: "28ebabbb-df4d-496d-95bf-0a6e07b92520"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:17.430241 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.430230    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ebabbb-df4d-496d-95bf-0a6e07b92520-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:17.431919 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.431899    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls" pod "28ebabbb-df4d-496d-95bf-0a6e07b92520" (UID: "28ebabbb-df4d-496d-95bf-0a6e07b92520"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:17.432065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.432020    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-metrics-certs" pod "28ebabbb-df4d-496d-95bf-0a6e07b92520" (UID: "28ebabbb-df4d-496d-95bf-0a6e07b92520"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:17.432126 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.432020    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ebabbb-df4d-496d-95bf-0a6e07b92520-kube-api-access-bxhsr" pod "28ebabbb-df4d-496d-95bf-0a6e07b92520" (UID: "28ebabbb-df4d-496d-95bf-0a6e07b92520"). InnerVolumeSpecName "kube-api-access-bxhsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:17.530693 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.530661    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:17.530693 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.530687    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ebabbb-df4d-496d-95bf-0a6e07b92520-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:17.530693 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.530697    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhsr\" (UniqueName: \"kubernetes.io/projected/28ebabbb-df4d-496d-95bf-0a6e07b92520-kube-api-access-bxhsr\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:17.669891 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.669854    4837 generic.go:356] "Generic (PLEG): container finished" podID="28ebabbb-df4d-496d-95bf-0a6e07b92520" containerID="480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092" exitCode=0
Nov 01 23:13:17.670081 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.669905    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"
Nov 01 23:13:17.670081 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.669931    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d" event={"ID":"28ebabbb-df4d-496d-95bf-0a6e07b92520","Type":"ContainerDied","Data":"480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092"}
Nov 01 23:13:17.670081 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.669982    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d" event={"ID":"28ebabbb-df4d-496d-95bf-0a6e07b92520","Type":"ContainerDied","Data":"d49a2e50e3108c0e6e8b65d930ccb3bbe18a9c7ab6dd5a8b5d29c71a05d63ee1"}
Nov 01 23:13:17.670081 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.670010    4837 scope.go:122] "RemoveContainer" containerID="480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092"
Nov 01 23:13:17.679922 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.679901    4837 scope.go:122] "RemoveContainer" containerID="480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092"
Nov 01 23:13:17.680199 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:17.680181    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092\": container with ID starting with 480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092 not found: ID does not exist" containerID="480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092"
Nov 01 23:13:17.680268 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.680211    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092"} err="failed to get container status \"480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092\": rpc error: code = NotFound desc = could not find container \"480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092\": container with ID starting with 480d908ea4b77e3e829f6d817ecc6686c6f5147a868836c366b10d6d7afbd092 not found: ID does not exist"
Nov 01 23:13:17.823830 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.823791    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"]
Nov 01 23:13:17.825885 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.825859    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8cc47a-5bf4555bbf-zgc6d"]
Nov 01 23:13:17.999430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:17.999356    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:13:18.524858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:18.524827    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ebabbb-df4d-496d-95bf-0a6e07b92520" path="/var/lib/kubelet/pods/28ebabbb-df4d-496d-95bf-0a6e07b92520/volumes"
Nov 01 23:13:18.811940 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:18.811863    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:13:20.095352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.095316    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"]
Nov 01 23:13:20.095812 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.095530    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy" containerID="cri-o://e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5" gracePeriod=30
Nov 01 23:13:20.901517 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901486    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"]
Nov 01 23:13:20.901879 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901867    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy"
Nov 01 23:13:20.901930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901879    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy"
Nov 01 23:13:20.901930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901890    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy"
Nov 01 23:13:20.901930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901897    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy"
Nov 01 23:13:20.901930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901910    4837 cpu_manager.go:414] "containerMap: removing container" podUID="bc50a1c4-0675-4939-8246-64c8d7cdbeec" containerName="maas-api"
Nov 01 23:13:20.901930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901915    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc50a1c4-0675-4939-8246-64c8d7cdbeec" containerName="maas-api"
Nov 01 23:13:20.901930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901928    4837 cpu_manager.go:414] "containerMap: removing container" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901933    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901941    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901946    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901960    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901965    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901973    4837 cpu_manager.go:414] "containerMap: removing container" podUID="fb963b29-0c1b-439c-a887-763eff140b30" containerName="maas-api"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901977    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb963b29-0c1b-439c-a887-763eff140b30" containerName="maas-api"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901983    4837 cpu_manager.go:414] "containerMap: removing container" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901987    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901994    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1236add6-9446-4069-b520-18789338dd10" containerName="main"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.901998    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1236add6-9446-4069-b520-18789338dd10" containerName="main"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902007    4837 cpu_manager.go:414] "containerMap: removing container" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902012    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902018    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14" containerName="payload-processing"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902022    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14" containerName="payload-processing"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902030    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e17dc971-5864-4d0a-807e-78e593e3fec1" containerName="maas-api"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902034    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17dc971-5864-4d0a-807e-78e593e3fec1" containerName="maas-api"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902057    4837 cpu_manager.go:414] "containerMap: removing container" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902061    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902068    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902072    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902080    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902084    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902091    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902096    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902103    4837 cpu_manager.go:414] "containerMap: removing container" podUID="28ebabbb-df4d-496d-95bf-0a6e07b92520" containerName="maas-api"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902107    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ebabbb-df4d-496d-95bf-0a6e07b92520" containerName="maas-api"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902114    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9358a5c7-3313-4dee-b5e4-d0ac7a810fde" containerName="maas-api"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902119    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9358a5c7-3313-4dee-b5e4-d0ac7a810fde" containerName="maas-api"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902128    4837 cpu_manager.go:414] "containerMap: removing container" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902132    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902138    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0ab1b3f3-5281-4815-87cd-df2dbea8b90c" containerName="payload-processing"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902143    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab1b3f3-5281-4815-87cd-df2dbea8b90c" containerName="payload-processing"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902151    4837 cpu_manager.go:414] "containerMap: removing container" podUID="7afb93f5-fcee-4434-bbfd-e297f8ea86cb" containerName="payload-processing"
Nov 01 23:13:20.902139 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902155    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7afb93f5-fcee-4434-bbfd-e297f8ea86cb" containerName="payload-processing"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902162    4837 cpu_manager.go:414] "containerMap: removing container" podUID="34b4e706-377c-43f8-ae38-8a610dcb0446" containerName="main"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902166    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b4e706-377c-43f8-ae38-8a610dcb0446" containerName="main"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902221    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbe9044-faad-4c63-bc0d-292c2e910a2d" containerName="main"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902230    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902236    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb963b29-0c1b-439c-a887-763eff140b30" containerName="maas-api"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902242    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902247    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1236add6-9446-4069-b520-18789338dd10" containerName="main"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902254    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17dc971-5864-4d0a-807e-78e593e3fec1" containerName="maas-api"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902260    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4aa13c-e4dc-48ad-ab95-127ab1a6bb14" containerName="payload-processing"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902265    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5463180-2162-4d32-b896-2ddd39e1b170" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902270    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ac7195-82b7-4b1a-99de-21e23225824b" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902276    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902282    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0165cb48-bf04-4815-a668-31a3af4f64ed" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902287    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ebabbb-df4d-496d-95bf-0a6e07b92520" containerName="maas-api"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902292    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9358a5c7-3313-4dee-b5e4-d0ac7a810fde" containerName="maas-api"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902297    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902301    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="629d787b-d1d4-4102-90dc-2c095dd59a56" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902305    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab1b3f3-5281-4815-87cd-df2dbea8b90c" containerName="payload-processing"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902311    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="7afb93f5-fcee-4434-bbfd-e297f8ea86cb" containerName="payload-processing"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902315    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b4e706-377c-43f8-ae38-8a610dcb0446" containerName="main"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902321    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4835c74-5ce8-4b54-8c88-49aa8b706df0" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902326    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902331    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc50a1c4-0675-4939-8246-64c8d7cdbeec" containerName="maas-api"
Nov 01 23:13:20.903194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.902336    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="828731a6-88ec-40b3-ac92-e57431e8dab8" containerName="istio-proxy"
Nov 01 23:13:20.906895 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.906875    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:20.961186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.961161    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:13:20.961437 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.961411    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:13:20.961578 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.961468    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:13:20.963762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.963739    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"]
Nov 01 23:13:20.968999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.968976    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:20.969112 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.969026    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:20.969177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:20.969109    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thp6m\" (UniqueName: \"kubernetes.io/projected/23b2cdc9-c815-4039-98f3-67097f743253-kube-api-access-thp6m\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:21.007577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.007560    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:13:21.069772 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.069747    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:21.069884 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.069780    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thp6m\" (UniqueName: \"kubernetes.io/projected/23b2cdc9-c815-4039-98f3-67097f743253-kube-api-access-thp6m\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:21.069884 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.069856    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:21.070366 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.070349    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:21.072089 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.072059    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:21.123247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.123230    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thp6m\" (UniqueName: \"kubernetes.io/projected/23b2cdc9-c815-4039-98f3-67097f743253-kube-api-access-thp6m\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:21.220494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.220472    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:21.585636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.585606    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"]
Nov 01 23:13:21.587388 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:13:21.587355    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b2cdc9_c815_4039_98f3_67097f743253.slice/crio-642e40724d32341ab845fa319602fdcda751d523bdecfb8a40e9a69878c9bc8e WatchSource:0}: Error finding container 642e40724d32341ab845fa319602fdcda751d523bdecfb8a40e9a69878c9bc8e: Status 404 returned error can't find the container with id 642e40724d32341ab845fa319602fdcda751d523bdecfb8a40e9a69878c9bc8e
Nov 01 23:13:21.691761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:21.691732    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs" event={"ID":"23b2cdc9-c815-4039-98f3-67097f743253","Type":"ContainerStarted","Data":"642e40724d32341ab845fa319602fdcda751d523bdecfb8a40e9a69878c9bc8e"}
Nov 01 23:13:22.697640 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:22.697612    4837 generic.go:356] "Generic (PLEG): container finished" podID="23b2cdc9-c815-4039-98f3-67097f743253" containerID="80195ddcc9f463b84817ec394179381cf2c6ca627bdf7320b91e0662d4b3abcd" exitCode=0
Nov 01 23:13:22.698053 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:22.697666    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs" event={"ID":"23b2cdc9-c815-4039-98f3-67097f743253","Type":"ContainerDied","Data":"80195ddcc9f463b84817ec394179381cf2c6ca627bdf7320b91e0662d4b3abcd"}
Nov 01 23:13:23.838482 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:23.838456    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:24.000213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.000145    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-xks-ca\") pod \"23b2cdc9-c815-4039-98f3-67097f743253\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") "
Nov 01 23:13:24.000345 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.000264    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b2cdc9-c815-4039-98f3-67097f743253-kube-api-access-thp6m\" (UniqueName: \"kubernetes.io/projected/23b2cdc9-c815-4039-98f3-67097f743253-kube-api-access-thp6m\") pod \"23b2cdc9-c815-4039-98f3-67097f743253\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") "
Nov 01 23:13:24.000400 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.000384    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-openshift-ca\") pod \"23b2cdc9-c815-4039-98f3-67097f743253\" (UID: \"23b2cdc9-c815-4039-98f3-67097f743253\") "
Nov 01 23:13:24.000694 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.000666    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-openshift-ca" pod "23b2cdc9-c815-4039-98f3-67097f743253" (UID: "23b2cdc9-c815-4039-98f3-67097f743253"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:24.002337 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.002312    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-xks-ca" pod "23b2cdc9-c815-4039-98f3-67097f743253" (UID: "23b2cdc9-c815-4039-98f3-67097f743253"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:24.002451 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.002383    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b2cdc9-c815-4039-98f3-67097f743253-kube-api-access-thp6m" pod "23b2cdc9-c815-4039-98f3-67097f743253" (UID: "23b2cdc9-c815-4039-98f3-67097f743253"). InnerVolumeSpecName "kube-api-access-thp6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:24.101973 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.101943    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:24.101973 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.101969    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b2cdc9-c815-4039-98f3-67097f743253-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:24.101973 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.101980    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thp6m\" (UniqueName: \"kubernetes.io/projected/23b2cdc9-c815-4039-98f3-67097f743253-kube-api-access-thp6m\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:24.707375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.707346    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"
Nov 01 23:13:24.707536 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.707374    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs" event={"ID":"23b2cdc9-c815-4039-98f3-67097f743253","Type":"ContainerDied","Data":"642e40724d32341ab845fa319602fdcda751d523bdecfb8a40e9a69878c9bc8e"}
Nov 01 23:13:24.707536 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:24.707423    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="642e40724d32341ab845fa319602fdcda751d523bdecfb8a40e9a69878c9bc8e"
Nov 01 23:13:25.348646 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.348624    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:13:25.514938 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.514912    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-podinfo\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.514963    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istiod-ca-cert\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.514982    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-socket\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515017    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-certs\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515079    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-data\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515128 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515110    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-credential-socket\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515149    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-envoy\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515191    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-token\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-token\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515230    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-kube-api-access-5vsg8\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-kube-api-access-5vsg8\") pod \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\" (UID: \"83ebe69f-f0df-4388-b1d1-b15c1e43351e\") "
Nov 01 23:13:25.515441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515323    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-socket" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:25.515441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515429    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istiod-ca-cert" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:25.515711 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515463    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-credential-socket" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:25.515711 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515631    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-certs" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:25.515818 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515712    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.515818 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515732    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.515818 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515746    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.515818 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.515751    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-data" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:25.517652 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.517624    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-podinfo" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:13:25.518071 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.518032    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-kube-api-access-5vsg8" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "kube-api-access-5vsg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:25.518183 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.518127    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-envoy" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:25.518289 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.518269    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-token" pod "83ebe69f-f0df-4388-b1d1-b15c1e43351e" (UID: "83ebe69f-f0df-4388-b1d1-b15c1e43351e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:25.617283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.617252    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.617283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.617276    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.617283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.617285    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.617509 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.617295    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.617509 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.617304    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.617509 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.617312    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vsg8\" (UniqueName: \"kubernetes.io/projected/83ebe69f-f0df-4388-b1d1-b15c1e43351e-kube-api-access-5vsg8\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:25.714236 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.714201    4837 generic.go:356] "Generic (PLEG): container finished" podID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerID="e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5" exitCode=0
Nov 01 23:13:25.714380 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.714278    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"
Nov 01 23:13:25.714380 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.714283    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" event={"ID":"83ebe69f-f0df-4388-b1d1-b15c1e43351e","Type":"ContainerDied","Data":"e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5"}
Nov 01 23:13:25.714380 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.714326    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s" event={"ID":"83ebe69f-f0df-4388-b1d1-b15c1e43351e","Type":"ContainerDied","Data":"4b665cab346b65c91a04b41d390aa2bac900d6dec696703300c0ba471d2086f1"}
Nov 01 23:13:25.714380 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.714345    4837 scope.go:122] "RemoveContainer" containerID="e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5"
Nov 01 23:13:25.723990 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.723976    4837 scope.go:122] "RemoveContainer" containerID="fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b"
Nov 01 23:13:25.731803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.731784    4837 scope.go:122] "RemoveContainer" containerID="e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5"
Nov 01 23:13:25.732079 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:25.732029    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5\": container with ID starting with e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5 not found: ID does not exist" containerID="e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5"
Nov 01 23:13:25.732177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.732086    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5"} err="failed to get container status \"e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5\": rpc error: code = NotFound desc = could not find container \"e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5\": container with ID starting with e8d0aa0a257d90012c31987e4f2eb41cf805f57f48ea25afbf12e34eb5ae73d5 not found: ID does not exist"
Nov 01 23:13:25.732177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.732104    4837 scope.go:122] "RemoveContainer" containerID="fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b"
Nov 01 23:13:25.732336 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:25.732319    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b\": container with ID starting with fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b not found: ID does not exist" containerID="fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b"
Nov 01 23:13:25.732378 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.732344    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b"} err="failed to get container status \"fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b\": rpc error: code = NotFound desc = could not find container \"fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b\": container with ID starting with fc3b2421b8a527ca4921064d9356258aa2a565124ecb7645166cbeeecac1eb0b not found: ID does not exist"
Nov 01 23:13:25.830832 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.830701    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"]
Nov 01 23:13:25.831418 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.830975    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy" containerID="cri-o://69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a" gracePeriod=30
Nov 01 23:13:25.877701 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.877667    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"]
Nov 01 23:13:25.884772 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:25.884747    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cf46c7-openshift-default-589b86cc75-67t9s"]
Nov 01 23:13:26.525117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.525088    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" path="/var/lib/kubelet/pods/83ebe69f-f0df-4388-b1d1-b15c1e43351e/volumes"
Nov 01 23:13:26.854601 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854523    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"]
Nov 01 23:13:26.854929 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854917    4837 cpu_manager.go:414] "containerMap: removing container" podUID="23b2cdc9-c815-4039-98f3-67097f743253" containerName="revoke-keys"
Nov 01 23:13:26.854978 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854931    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b2cdc9-c815-4039-98f3-67097f743253" containerName="revoke-keys"
Nov 01 23:13:26.854978 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854963    4837 cpu_manager.go:414] "containerMap: removing container" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy"
Nov 01 23:13:26.854978 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854968    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy"
Nov 01 23:13:26.854978 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854976    4837 cpu_manager.go:414] "containerMap: removing container" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy"
Nov 01 23:13:26.855121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854982    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy"
Nov 01 23:13:26.855121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854990    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy"
Nov 01 23:13:26.855121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.854996    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2f5e83-f48d-422e-a12c-65942e5ceb5f" containerName="istio-proxy"
Nov 01 23:13:26.855121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.855069    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b2cdc9-c815-4039-98f3-67097f743253" containerName="revoke-keys"
Nov 01 23:13:26.855121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.855079    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2baff4f5-c9e3-4860-9a29-2b36a1e09547" containerName="istio-proxy"
Nov 01 23:13:26.855121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.855089    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy"
Nov 01 23:13:26.855121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.855095    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ebe69f-f0df-4388-b1d1-b15c1e43351e" containerName="istio-proxy"
Nov 01 23:13:26.859702 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.859684    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:26.910590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.910567    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:13:26.910590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.910576    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:13:26.910790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.910568    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:13:26.956736 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.956710    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:13:26.958519 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:26.958500    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"]
Nov 01 23:13:27.029563 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.029537    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zmc\" (UniqueName: \"kubernetes.io/projected/496c2959-267b-4a5f-89c5-314bfb42874b-kube-api-access-46zmc\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.029680 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.029593    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.029680 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.029614    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.131094 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.130997    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.131094 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.131036    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.131264 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.131157    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zmc\" (UniqueName: \"kubernetes.io/projected/496c2959-267b-4a5f-89c5-314bfb42874b-kube-api-access-46zmc\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.131724 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.131697    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.133364 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.133345    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.184367 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.184344    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zmc\" (UniqueName: \"kubernetes.io/projected/496c2959-267b-4a5f-89c5-314bfb42874b-kube-api-access-46zmc\") pod \"maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.475516 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.475481    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:27.837618 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:27.837594    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"]
Nov 01 23:13:27.840060 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:13:27.840016    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496c2959_267b_4a5f_89c5_314bfb42874b.slice/crio-8cc8df4f7dd4e8ff71b4eef48996c6d6f51740d8c3dd65d73499074678e3724c WatchSource:0}: Error finding container 8cc8df4f7dd4e8ff71b4eef48996c6d6f51740d8c3dd65d73499074678e3724c: Status 404 returned error can't find the container with id 8cc8df4f7dd4e8ff71b4eef48996c6d6f51740d8c3dd65d73499074678e3724c
Nov 01 23:13:28.731873 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:28.731844    4837 generic.go:356] "Generic (PLEG): container finished" podID="496c2959-267b-4a5f-89c5-314bfb42874b" containerID="1fd8eded7261a499ba65abb8fc1c9ecea05f20cbde9960781fd08c30459d21ef" exitCode=0
Nov 01 23:13:28.731989 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:28.731897    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm" event={"ID":"496c2959-267b-4a5f-89c5-314bfb42874b","Type":"ContainerDied","Data":"1fd8eded7261a499ba65abb8fc1c9ecea05f20cbde9960781fd08c30459d21ef"}
Nov 01 23:13:28.731989 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:28.731935    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm" event={"ID":"496c2959-267b-4a5f-89c5-314bfb42874b","Type":"ContainerStarted","Data":"8cc8df4f7dd4e8ff71b4eef48996c6d6f51740d8c3dd65d73499074678e3724c"}
Nov 01 23:13:29.873824 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:29.873802    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:29.958638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:29.958612    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496c2959-267b-4a5f-89c5-314bfb42874b-kube-api-access-46zmc\" (UniqueName: \"kubernetes.io/projected/496c2959-267b-4a5f-89c5-314bfb42874b-kube-api-access-46zmc\") pod \"496c2959-267b-4a5f-89c5-314bfb42874b\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") "
Nov 01 23:13:29.958782 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:29.958670    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-xks-ca\") pod \"496c2959-267b-4a5f-89c5-314bfb42874b\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") "
Nov 01 23:13:29.958782 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:29.958737    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-openshift-ca\") pod \"496c2959-267b-4a5f-89c5-314bfb42874b\" (UID: \"496c2959-267b-4a5f-89c5-314bfb42874b\") "
Nov 01 23:13:29.959105 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:29.959083    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-openshift-ca" pod "496c2959-267b-4a5f-89c5-314bfb42874b" (UID: "496c2959-267b-4a5f-89c5-314bfb42874b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:29.960618 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:29.960598    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496c2959-267b-4a5f-89c5-314bfb42874b-kube-api-access-46zmc" pod "496c2959-267b-4a5f-89c5-314bfb42874b" (UID: "496c2959-267b-4a5f-89c5-314bfb42874b"). InnerVolumeSpecName "kube-api-access-46zmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:29.960709 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:29.960674    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-xks-ca" pod "496c2959-267b-4a5f-89c5-314bfb42874b" (UID: "496c2959-267b-4a5f-89c5-314bfb42874b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:30.060353 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:30.060276    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zmc\" (UniqueName: \"kubernetes.io/projected/496c2959-267b-4a5f-89c5-314bfb42874b-kube-api-access-46zmc\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:30.060353 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:30.060304    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:30.060353 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:30.060313    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496c2959-267b-4a5f-89c5-314bfb42874b-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:30.743415 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:30.743383    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"
Nov 01 23:13:30.743583 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:30.743386    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm" event={"ID":"496c2959-267b-4a5f-89c5-314bfb42874b","Type":"ContainerDied","Data":"8cc8df4f7dd4e8ff71b4eef48996c6d6f51740d8c3dd65d73499074678e3724c"}
Nov 01 23:13:30.743583 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:30.743503    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cc8df4f7dd4e8ff71b4eef48996c6d6f51740d8c3dd65d73499074678e3724c"
Nov 01 23:13:31.083728 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.083699    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 23:13:31.272317 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272288    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-envoy\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272473 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272347    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-podinfo\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272473 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272373    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-kube-api-access-t4bbc\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-kube-api-access-t4bbc\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272473 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272403    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-token\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-token\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272473 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272449    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-socket\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272724 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272530    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-data\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272724 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272602    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-certs\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272724 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272638    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-credential-socket\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272724 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272685    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0049fa-0562-4103-8df1-ba15d48f58ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0049fa-0562-4103-8df1-ba15d48f58ba-istiod-ca-cert\") pod \"cf0049fa-0562-4103-8df1-ba15d48f58ba\" (UID: \"cf0049fa-0562-4103-8df1-ba15d48f58ba\") "
Nov 01 23:13:31.272970 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272720    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-socket" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:31.272970 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272802    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-data" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:31.272970 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272875    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-certs" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:31.272970 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.272945    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-credential-socket" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:31.273238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.273132    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0049fa-0562-4103-8df1-ba15d48f58ba-istiod-ca-cert" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:31.273238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.273135    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.273238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.273177    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.273238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.273195    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.273238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.273210    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.274636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.274600    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-envoy" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:31.274750 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.274616    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-token" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:31.274750 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.274671    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-kube-api-access-t4bbc" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "kube-api-access-t4bbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:31.274867 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.274846    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-podinfo" pod "cf0049fa-0562-4103-8df1-ba15d48f58ba" (UID: "cf0049fa-0562-4103-8df1-ba15d48f58ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:13:31.280268 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.280243    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"]
Nov 01 23:13:31.294129 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.294108    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"]
Nov 01 23:13:31.381539 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.375323    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0049fa-0562-4103-8df1-ba15d48f58ba-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.381539 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.375391    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.381539 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.375431    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.381539 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.375452    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4bbc\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-kube-api-access-t4bbc\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.381539 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.375474    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0049fa-0562-4103-8df1-ba15d48f58ba-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:31.749269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.749239    4837 generic.go:356] "Generic (PLEG): container finished" podID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerID="69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a" exitCode=0
Nov 01 23:13:31.749435 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.749302    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"
Nov 01 23:13:31.749435 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.749330    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" event={"ID":"cf0049fa-0562-4103-8df1-ba15d48f58ba","Type":"ContainerDied","Data":"69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a"}
Nov 01 23:13:31.749435 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.749386    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns" event={"ID":"cf0049fa-0562-4103-8df1-ba15d48f58ba","Type":"ContainerDied","Data":"a17ce1b5510087a6bcb927140f21204f8ca43a81eb429d5eee65c11b15df9e8e"}
Nov 01 23:13:31.749435 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.749408    4837 scope.go:122] "RemoveContainer" containerID="69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a"
Nov 01 23:13:31.759072 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.759052    4837 scope.go:122] "RemoveContainer" containerID="6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954"
Nov 01 23:13:31.767225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.767210    4837 scope.go:122] "RemoveContainer" containerID="69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a"
Nov 01 23:13:31.767487 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:31.767468    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a\": container with ID starting with 69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a not found: ID does not exist" containerID="69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a"
Nov 01 23:13:31.767535 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.767496    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a"} err="failed to get container status \"69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a\": rpc error: code = NotFound desc = could not find container \"69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a\": container with ID starting with 69e9379d66dcbb7488ebefe2428c58f8f7d67cdf93a65da8c525b8b24473727a not found: ID does not exist"
Nov 01 23:13:31.767535 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.767515    4837 scope.go:122] "RemoveContainer" containerID="6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954"
Nov 01 23:13:31.767729 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:31.767711    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954\": container with ID starting with 6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954 not found: ID does not exist" containerID="6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954"
Nov 01 23:13:31.767787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.767737    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954"} err="failed to get container status \"6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954\": rpc error: code = NotFound desc = could not find container \"6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954\": container with ID starting with 6c7c4ac06a0a4d974f5bd6755c95003ef8cbd73dd17372a6853fc60c7c29f954 not found: ID does not exist"
Nov 01 23:13:31.903805 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.903779    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"]
Nov 01 23:13:31.949724 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:31.949697    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8cc47a-openshift-default-56f7766f7-97nns"]
Nov 01 23:13:32.523864 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:32.523833    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" path="/var/lib/kubelet/pods/cf0049fa-0562-4103-8df1-ba15d48f58ba/volumes"
Nov 01 23:13:38.959104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:38.959070    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:13:38.964579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:38.964558    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:13:43.013829 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:43.013798    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:13:46.158154 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.158122    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"]
Nov 01 23:13:46.204242 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.204210    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8bac2f-8qfxs"]
Nov 01 23:13:46.301143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.301089    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main" containerID="cri-o://227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c" gracePeriod=45
Nov 01 23:13:46.312501 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.312459    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main" containerID="cri-o://95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b" gracePeriod=45
Nov 01 23:13:46.524987 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.524957    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b2cdc9-c815-4039-98f3-67097f743253" path="/var/lib/kubelet/pods/23b2cdc9-c815-4039-98f3-67097f743253/volumes"
Nov 01 23:13:46.567585 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.567543    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 23:13:46.591475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.591456    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 23:13:46.601850 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.601831    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-tmp-dir\") pod \"c74d4432-900c-4bc6-b703-bcae0e67c109\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") "
Nov 01 23:13:46.601935 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.601871    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74d4432-900c-4bc6-b703-bcae0e67c109-tls-certs\" (UniqueName: \"kubernetes.io/secret/c74d4432-900c-4bc6-b703-bcae0e67c109-tls-certs\") pod \"c74d4432-900c-4bc6-b703-bcae0e67c109\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") "
Nov 01 23:13:46.601935 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.601923    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74d4432-900c-4bc6-b703-bcae0e67c109-kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/c74d4432-900c-4bc6-b703-bcae0e67c109-kube-api-access-ksvwh\") pod \"c74d4432-900c-4bc6-b703-bcae0e67c109\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") "
Nov 01 23:13:46.602034 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.601951    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-dshm\") pod \"c74d4432-900c-4bc6-b703-bcae0e67c109\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") "
Nov 01 23:13:46.602034 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.602026    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-model-cache\") pod \"c74d4432-900c-4bc6-b703-bcae0e67c109\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") "
Nov 01 23:13:46.602149 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.602081    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-tmp-dir" pod "c74d4432-900c-4bc6-b703-bcae0e67c109" (UID: "c74d4432-900c-4bc6-b703-bcae0e67c109"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.602149 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.602095    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-home\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-home\") pod \"c74d4432-900c-4bc6-b703-bcae0e67c109\" (UID: \"c74d4432-900c-4bc6-b703-bcae0e67c109\") "
Nov 01 23:13:46.602284 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.602266    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-model-cache" pod "c74d4432-900c-4bc6-b703-bcae0e67c109" (UID: "c74d4432-900c-4bc6-b703-bcae0e67c109"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.602418 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.602395    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-home" pod "c74d4432-900c-4bc6-b703-bcae0e67c109" (UID: "c74d4432-900c-4bc6-b703-bcae0e67c109"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.602612 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.602557    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.602612 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.602579    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.602612 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.602593    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.604126 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.604104    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-dshm" pod "c74d4432-900c-4bc6-b703-bcae0e67c109" (UID: "c74d4432-900c-4bc6-b703-bcae0e67c109"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.604196 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.604172    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74d4432-900c-4bc6-b703-bcae0e67c109-tls-certs" pod "c74d4432-900c-4bc6-b703-bcae0e67c109" (UID: "c74d4432-900c-4bc6-b703-bcae0e67c109"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:46.604235 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.604199    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74d4432-900c-4bc6-b703-bcae0e67c109-kube-api-access-ksvwh" pod "c74d4432-900c-4bc6-b703-bcae0e67c109" (UID: "c74d4432-900c-4bc6-b703-bcae0e67c109"). InnerVolumeSpecName "kube-api-access-ksvwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:46.703256 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703021    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d01282-ad46-4cc0-bf80-3113fad5392e-kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/53d01282-ad46-4cc0-bf80-3113fad5392e-kube-api-access-824ts\") pod \"53d01282-ad46-4cc0-bf80-3113fad5392e\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") "
Nov 01 23:13:46.703256 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703107    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-dshm\") pod \"53d01282-ad46-4cc0-bf80-3113fad5392e\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") "
Nov 01 23:13:46.703256 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703241    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-model-cache\") pod \"53d01282-ad46-4cc0-bf80-3113fad5392e\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") "
Nov 01 23:13:46.703542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703304    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-tmp-dir\") pod \"53d01282-ad46-4cc0-bf80-3113fad5392e\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") "
Nov 01 23:13:46.703542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703376    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53d01282-ad46-4cc0-bf80-3113fad5392e-tls-certs\" (UniqueName: \"kubernetes.io/secret/53d01282-ad46-4cc0-bf80-3113fad5392e-tls-certs\") pod \"53d01282-ad46-4cc0-bf80-3113fad5392e\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") "
Nov 01 23:13:46.703542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703431    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-home\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-home\") pod \"53d01282-ad46-4cc0-bf80-3113fad5392e\" (UID: \"53d01282-ad46-4cc0-bf80-3113fad5392e\") "
Nov 01 23:13:46.703714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703565    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-tmp-dir" pod "53d01282-ad46-4cc0-bf80-3113fad5392e" (UID: "53d01282-ad46-4cc0-bf80-3113fad5392e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.703714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703603    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-model-cache" pod "53d01282-ad46-4cc0-bf80-3113fad5392e" (UID: "53d01282-ad46-4cc0-bf80-3113fad5392e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.703854 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703833    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-home" pod "53d01282-ad46-4cc0-bf80-3113fad5392e" (UID: "53d01282-ad46-4cc0-bf80-3113fad5392e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.703920 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703873    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.703920 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703894    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c74d4432-900c-4bc6-b703-bcae0e67c109-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.703920 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703911    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvwh\" (UniqueName: \"kubernetes.io/projected/c74d4432-900c-4bc6-b703-bcae0e67c109-kube-api-access-ksvwh\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.704136 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703926    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c74d4432-900c-4bc6-b703-bcae0e67c109-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.704136 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.703949    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.705291 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.705264    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d01282-ad46-4cc0-bf80-3113fad5392e-kube-api-access-824ts" pod "53d01282-ad46-4cc0-bf80-3113fad5392e" (UID: "53d01282-ad46-4cc0-bf80-3113fad5392e"). InnerVolumeSpecName "kube-api-access-824ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:46.705291 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.705273    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-dshm" pod "53d01282-ad46-4cc0-bf80-3113fad5392e" (UID: "53d01282-ad46-4cc0-bf80-3113fad5392e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.705920 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.705900    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53d01282-ad46-4cc0-bf80-3113fad5392e-tls-certs" pod "53d01282-ad46-4cc0-bf80-3113fad5392e" (UID: "53d01282-ad46-4cc0-bf80-3113fad5392e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:46.805335 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.805301    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53d01282-ad46-4cc0-bf80-3113fad5392e-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805335 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.805326    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805335 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.805336    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824ts\" (UniqueName: \"kubernetes.io/projected/53d01282-ad46-4cc0-bf80-3113fad5392e-kube-api-access-824ts\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805335 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.805345    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53d01282-ad46-4cc0-bf80-3113fad5392e-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.830024 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.829989    4837 generic.go:356] "Generic (PLEG): container finished" podID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerID="227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c" exitCode=0
Nov 01 23:13:46.830296 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.830100    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" event={"ID":"c74d4432-900c-4bc6-b703-bcae0e67c109","Type":"ContainerDied","Data":"227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c"}
Nov 01 23:13:46.830296 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.830035    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"
Nov 01 23:13:46.830296 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.830165    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6" event={"ID":"c74d4432-900c-4bc6-b703-bcae0e67c109","Type":"ContainerDied","Data":"00d2e29188fc5f6e76450e3fa68a36209d36616662822035e37003e84c1edf15"}
Nov 01 23:13:46.830296 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.830191    4837 scope.go:122] "RemoveContainer" containerID="227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c"
Nov 01 23:13:46.832095 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.831937    4837 generic.go:356] "Generic (PLEG): container finished" podID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerID="95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b" exitCode=0
Nov 01 23:13:46.832095 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.832008    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"
Nov 01 23:13:46.832095 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.832023    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" event={"ID":"53d01282-ad46-4cc0-bf80-3113fad5392e","Type":"ContainerDied","Data":"95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b"}
Nov 01 23:13:46.832095 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.832066    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7" event={"ID":"53d01282-ad46-4cc0-bf80-3113fad5392e","Type":"ContainerDied","Data":"25ce110e10f10de36d30d38248c03e8dcfc8877bac53a6aa7c6a0539c86a6974"}
Nov 01 23:13:46.842445 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.842425    4837 scope.go:122] "RemoveContainer" containerID="227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c"
Nov 01 23:13:46.842780 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:46.842752    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c\": container with ID starting with 227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c not found: ID does not exist" containerID="227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c"
Nov 01 23:13:46.842846 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.842794    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c"} err="failed to get container status \"227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c\": rpc error: code = NotFound desc = could not find container \"227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c\": container with ID starting with 227957fa9fd8f5f564d7b9f22c02bd5ac9783928cda260fb32edc59e8b75cf1c not found: ID does not exist"
Nov 01 23:13:46.842846 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.842820    4837 scope.go:122] "RemoveContainer" containerID="95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b"
Nov 01 23:13:46.854077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.854027    4837 scope.go:122] "RemoveContainer" containerID="95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b"
Nov 01 23:13:46.854348 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:46.854330    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b\": container with ID starting with 95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b not found: ID does not exist" containerID="95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b"
Nov 01 23:13:46.854429 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.854357    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b"} err="failed to get container status \"95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b\": rpc error: code = NotFound desc = could not find container \"95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b\": container with ID starting with 95c0dfb9f8bd2eaa6c30922c118ff018b31ea1ff42efd11603190c201309d54b not found: ID does not exist"
Nov 01 23:13:46.984797 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.984724    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"]
Nov 01 23:13:46.986197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:46.986171    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-2-w2-c768f3-kserve-5899c8b699-pgzs6"]
Nov 01 23:13:47.088927 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:47.088901    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"]
Nov 01 23:13:47.093783 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:47.093764    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/premium-w2-c768f3-kserve-747bf497c7-hnmf7"]
Nov 01 23:13:48.523268 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:48.523187    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" path="/var/lib/kubelet/pods/53d01282-ad46-4cc0-bf80-3113fad5392e/volumes"
Nov 01 23:13:48.523769 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:48.523744    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" path="/var/lib/kubelet/pods/c74d4432-900c-4bc6-b703-bcae0e67c109/volumes"
Nov 01 23:13:48.810877 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:13:48.810800    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:13:53.397520 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397492    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"]
Nov 01 23:13:53.397918 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397857    4837 cpu_manager.go:414] "containerMap: removing container" podUID="496c2959-267b-4a5f-89c5-314bfb42874b" containerName="revoke-keys"
Nov 01 23:13:53.397918 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397867    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496c2959-267b-4a5f-89c5-314bfb42874b" containerName="revoke-keys"
Nov 01 23:13:53.397918 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397881    4837 cpu_manager.go:414] "containerMap: removing container" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main"
Nov 01 23:13:53.397918 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397885    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main"
Nov 01 23:13:53.397918 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397898    4837 cpu_manager.go:414] "containerMap: removing container" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy"
Nov 01 23:13:53.397918 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397903    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy"
Nov 01 23:13:53.398153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397924    4837 cpu_manager.go:414] "containerMap: removing container" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main"
Nov 01 23:13:53.398153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397930    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main"
Nov 01 23:13:53.398153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397939    4837 cpu_manager.go:414] "containerMap: removing container" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy"
Nov 01 23:13:53.398153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397944    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy"
Nov 01 23:13:53.398153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.397996    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74d4432-900c-4bc6-b703-bcae0e67c109" containerName="main"
Nov 01 23:13:53.398153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.398005    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy"
Nov 01 23:13:53.398153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.398011    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="496c2959-267b-4a5f-89c5-314bfb42874b" containerName="revoke-keys"
Nov 01 23:13:53.398153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.398018    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d01282-ad46-4cc0-bf80-3113fad5392e" containerName="main"
Nov 01 23:13:53.402542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.402524    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.454626 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.454604    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:13:53.461183 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.461150    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7qb\" (UniqueName: \"kubernetes.io/projected/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-kube-api-access-4n7qb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.461288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.461201    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.461356 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.461336    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.500943 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.500916    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:13:53.500943 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.500931    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:13:53.501142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.500914    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:13:53.502597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.502577    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"]
Nov 01 23:13:53.562147 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.562124    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7qb\" (UniqueName: \"kubernetes.io/projected/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-kube-api-access-4n7qb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.562239 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.562155    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.562239 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.562185    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.562892 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.562871    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.564635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.564618    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.616054 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.616012    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7qb\" (UniqueName: \"kubernetes.io/projected/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-kube-api-access-4n7qb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:53.715103 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:53.715081    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:54.079989 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:54.079962    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"]
Nov 01 23:13:54.082191 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:13:54.082157    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f69b1d7_7b72_4cf9_baf5_f13cebeaa443.slice/crio-43df0c92fad7cde62b2f6ff9be10d943e05045f34e9193ecbdff7d82f4f09831 WatchSource:0}: Error finding container 43df0c92fad7cde62b2f6ff9be10d943e05045f34e9193ecbdff7d82f4f09831: Status 404 returned error can't find the container with id 43df0c92fad7cde62b2f6ff9be10d943e05045f34e9193ecbdff7d82f4f09831
Nov 01 23:13:54.873197 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:54.873169    4837 generic.go:356] "Generic (PLEG): container finished" podID="0f69b1d7-7b72-4cf9-baf5-f13cebeaa443" containerID="821e543429e985f9da89c13924ff79e37dc3ed1b07ea8c048d4b7f90e41c8f65" exitCode=0
Nov 01 23:13:54.873525 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:54.873237    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6" event={"ID":"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443","Type":"ContainerDied","Data":"821e543429e985f9da89c13924ff79e37dc3ed1b07ea8c048d4b7f90e41c8f65"}
Nov 01 23:13:54.873525 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:54.873308    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6" event={"ID":"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443","Type":"ContainerStarted","Data":"43df0c92fad7cde62b2f6ff9be10d943e05045f34e9193ecbdff7d82f4f09831"}
Nov 01 23:13:55.558187 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.558154    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"]
Nov 01 23:13:55.558886 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.558865    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0049fa-0562-4103-8df1-ba15d48f58ba" containerName="istio-proxy"
Nov 01 23:13:55.562317 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.562300    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.615930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.615898    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"]
Nov 01 23:13:55.679131 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.679098    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.679279 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.679139    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.679279 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.679260    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmsd\" (UniqueName: \"kubernetes.io/projected/126de8c6-4972-48a1-8579-8407b9f33750-kube-api-access-htmsd\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.780562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.780533    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.780703 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.780589    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmsd\" (UniqueName: \"kubernetes.io/projected/126de8c6-4972-48a1-8579-8407b9f33750-kube-api-access-htmsd\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.780703 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.780634    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.781250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.781230    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.782852 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.782832    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.835432 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.835374    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmsd\" (UniqueName: \"kubernetes.io/projected/126de8c6-4972-48a1-8579-8407b9f33750-kube-api-access-htmsd\") pod \"maas-api-revoke-sub-f5b804f5-j6z9n\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:55.875199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:55.875163    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:56.016850 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.016824    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:56.084973 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.084946    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-xks-ca\") pod \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") "
Nov 01 23:13:56.085117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.084992    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-openshift-ca\") pod \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") "
Nov 01 23:13:56.085117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.085075    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-kube-api-access-4n7qb\" (UniqueName: \"kubernetes.io/projected/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-kube-api-access-4n7qb\") pod \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\" (UID: \"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443\") "
Nov 01 23:13:56.085402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.085379    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-openshift-ca" pod "0f69b1d7-7b72-4cf9-baf5-f13cebeaa443" (UID: "0f69b1d7-7b72-4cf9-baf5-f13cebeaa443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:56.086921 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.086874    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-xks-ca" pod "0f69b1d7-7b72-4cf9-baf5-f13cebeaa443" (UID: "0f69b1d7-7b72-4cf9-baf5-f13cebeaa443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:56.087011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.086993    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-kube-api-access-4n7qb" pod "0f69b1d7-7b72-4cf9-baf5-f13cebeaa443" (UID: "0f69b1d7-7b72-4cf9-baf5-f13cebeaa443"). InnerVolumeSpecName "kube-api-access-4n7qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:56.185972 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.185946    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:56.185972 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.185970    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:56.186176 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.185981    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7qb\" (UniqueName: \"kubernetes.io/projected/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443-kube-api-access-4n7qb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:56.248890 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.248866    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"]
Nov 01 23:13:56.251638 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:13:56.251611    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126de8c6_4972_48a1_8579_8407b9f33750.slice/crio-273128d8808788d730b0fdfa39baecf08fb0c15ffd46a8bf7efd212ebeff1ce5 WatchSource:0}: Error finding container 273128d8808788d730b0fdfa39baecf08fb0c15ffd46a8bf7efd212ebeff1ce5: Status 404 returned error can't find the container with id 273128d8808788d730b0fdfa39baecf08fb0c15ffd46a8bf7efd212ebeff1ce5
Nov 01 23:13:56.885656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.885632    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"
Nov 01 23:13:56.886154 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.885632    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6" event={"ID":"0f69b1d7-7b72-4cf9-baf5-f13cebeaa443","Type":"ContainerDied","Data":"43df0c92fad7cde62b2f6ff9be10d943e05045f34e9193ecbdff7d82f4f09831"}
Nov 01 23:13:56.886154 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.885756    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43df0c92fad7cde62b2f6ff9be10d943e05045f34e9193ecbdff7d82f4f09831"
Nov 01 23:13:56.887247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.887225    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n" event={"ID":"126de8c6-4972-48a1-8579-8407b9f33750","Type":"ContainerStarted","Data":"9beb4e4a27a254fc925f941e7ec258c061ae04b5d683b3f8f8a61a4f96861876"}
Nov 01 23:13:56.887378 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:56.887265    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n" event={"ID":"126de8c6-4972-48a1-8579-8407b9f33750","Type":"ContainerStarted","Data":"273128d8808788d730b0fdfa39baecf08fb0c15ffd46a8bf7efd212ebeff1ce5"}
Nov 01 23:13:57.003287 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:57.003242    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n" podStartSLOduration=2.003229673 podStartE2EDuration="2.003229673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:57.002971787 +0000 UTC m=+3018.592170850" watchObservedRunningTime="2026-11-01 23:13:57.003229673 +0000 UTC m=+3018.592428734"
Nov 01 23:13:57.640444 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:57.640412    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:13:57.894104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:57.893993    4837 generic.go:356] "Generic (PLEG): container finished" podID="126de8c6-4972-48a1-8579-8407b9f33750" containerID="9beb4e4a27a254fc925f941e7ec258c061ae04b5d683b3f8f8a61a4f96861876" exitCode=0
Nov 01 23:13:57.894665 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:57.894077    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n" event={"ID":"126de8c6-4972-48a1-8579-8407b9f33750","Type":"ContainerDied","Data":"9beb4e4a27a254fc925f941e7ec258c061ae04b5d683b3f8f8a61a4f96861876"}
Nov 01 23:13:59.036411 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.036383    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:59.112758 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.112733    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-xks-ca\") pod \"126de8c6-4972-48a1-8579-8407b9f33750\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") "
Nov 01 23:13:59.112895 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.112818    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-openshift-ca\") pod \"126de8c6-4972-48a1-8579-8407b9f33750\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") "
Nov 01 23:13:59.112936 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.112896    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126de8c6-4972-48a1-8579-8407b9f33750-kube-api-access-htmsd\" (UniqueName: \"kubernetes.io/projected/126de8c6-4972-48a1-8579-8407b9f33750-kube-api-access-htmsd\") pod \"126de8c6-4972-48a1-8579-8407b9f33750\" (UID: \"126de8c6-4972-48a1-8579-8407b9f33750\") "
Nov 01 23:13:59.113195 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.113174    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-openshift-ca" pod "126de8c6-4972-48a1-8579-8407b9f33750" (UID: "126de8c6-4972-48a1-8579-8407b9f33750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:59.114694 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.114667    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-xks-ca" pod "126de8c6-4972-48a1-8579-8407b9f33750" (UID: "126de8c6-4972-48a1-8579-8407b9f33750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:59.114784 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.114762    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126de8c6-4972-48a1-8579-8407b9f33750-kube-api-access-htmsd" pod "126de8c6-4972-48a1-8579-8407b9f33750" (UID: "126de8c6-4972-48a1-8579-8407b9f33750"). InnerVolumeSpecName "kube-api-access-htmsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:59.213981 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.213955    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htmsd\" (UniqueName: \"kubernetes.io/projected/126de8c6-4972-48a1-8579-8407b9f33750-kube-api-access-htmsd\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:59.213981 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.213979    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:59.214302 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.213988    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126de8c6-4972-48a1-8579-8407b9f33750-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:59.906808 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.906780    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"
Nov 01 23:13:59.906981 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.906777    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n" event={"ID":"126de8c6-4972-48a1-8579-8407b9f33750","Type":"ContainerDied","Data":"273128d8808788d730b0fdfa39baecf08fb0c15ffd46a8bf7efd212ebeff1ce5"}
Nov 01 23:13:59.906981 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:13:59.906927    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="273128d8808788d730b0fdfa39baecf08fb0c15ffd46a8bf7efd212ebeff1ce5"
Nov 01 23:14:04.895148 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:04.895115    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"]
Nov 01 23:14:09.168402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:09.168370    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"]
Nov 01 23:14:09.214554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:09.214525    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c768f3-n5wlm"]
Nov 01 23:14:09.895751 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:09.895715    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq" podUID="e5b26295-9a04-41fb-8c2d-7633791fd7f9" containerName="payload-processing" containerID="cri-o://ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240" gracePeriod=25
Nov 01 23:14:10.146323 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.146272    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 23:14:10.207376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.207352    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b26295-9a04-41fb-8c2d-7633791fd7f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b26295-9a04-41fb-8c2d-7633791fd7f9-plugins-config-volume\") pod \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\" (UID: \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\") "
Nov 01 23:14:10.207729 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.207516    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b26295-9a04-41fb-8c2d-7633791fd7f9-kube-api-access-2h298\" (UniqueName: \"kubernetes.io/projected/e5b26295-9a04-41fb-8c2d-7633791fd7f9-kube-api-access-2h298\") pod \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\" (UID: \"e5b26295-9a04-41fb-8c2d-7633791fd7f9\") "
Nov 01 23:14:10.207786 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.207744    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b26295-9a04-41fb-8c2d-7633791fd7f9-plugins-config-volume" pod "e5b26295-9a04-41fb-8c2d-7633791fd7f9" (UID: "e5b26295-9a04-41fb-8c2d-7633791fd7f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:10.209551 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.209526    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b26295-9a04-41fb-8c2d-7633791fd7f9-kube-api-access-2h298" pod "e5b26295-9a04-41fb-8c2d-7633791fd7f9" (UID: "e5b26295-9a04-41fb-8c2d-7633791fd7f9"). InnerVolumeSpecName "kube-api-access-2h298". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:10.243269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.243241    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"]
Nov 01 23:14:10.244600 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.244581    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-661e9f-rjsr6"]
Nov 01 23:14:10.309091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.309065    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h298\" (UniqueName: \"kubernetes.io/projected/e5b26295-9a04-41fb-8c2d-7633791fd7f9-kube-api-access-2h298\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:10.309091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.309087    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b26295-9a04-41fb-8c2d-7633791fd7f9-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:10.523303 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.523274    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f69b1d7-7b72-4cf9-baf5-f13cebeaa443" path="/var/lib/kubelet/pods/0f69b1d7-7b72-4cf9-baf5-f13cebeaa443/volumes"
Nov 01 23:14:10.523694 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.523682    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496c2959-267b-4a5f-89c5-314bfb42874b" path="/var/lib/kubelet/pods/496c2959-267b-4a5f-89c5-314bfb42874b/volumes"
Nov 01 23:14:10.958743 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.958714    4837 generic.go:356] "Generic (PLEG): container finished" podID="e5b26295-9a04-41fb-8c2d-7633791fd7f9" containerID="ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240" exitCode=0
Nov 01 23:14:10.958982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.958758    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"
Nov 01 23:14:10.958982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.958794    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq" event={"ID":"e5b26295-9a04-41fb-8c2d-7633791fd7f9","Type":"ContainerDied","Data":"ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240"}
Nov 01 23:14:10.958982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.958832    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq" event={"ID":"e5b26295-9a04-41fb-8c2d-7633791fd7f9","Type":"ContainerDied","Data":"e9e2a23f8db1a207b3e85f78de2a7c086a22293cf17d9064813c6bdb832e7030"}
Nov 01 23:14:10.958982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.958853    4837 scope.go:122] "RemoveContainer" containerID="ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240"
Nov 01 23:14:10.967675 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.967649    4837 scope.go:122] "RemoveContainer" containerID="ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240"
Nov 01 23:14:10.967909 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:14:10.967893    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240\": container with ID starting with ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240 not found: ID does not exist" containerID="ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240"
Nov 01 23:14:10.967955 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:10.967916    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240"} err="failed to get container status \"ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240\": rpc error: code = NotFound desc = could not find container \"ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240\": container with ID starting with ea618e858293fbf2aa93302e2e03b382928f16ae75f0c5609728975c64710240 not found: ID does not exist"
Nov 01 23:14:11.110408 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:11.110379    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"]
Nov 01 23:14:11.156310 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:11.156287    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-661e9f-6c9cc6b59c-xctxq"]
Nov 01 23:14:12.522907 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:12.522876    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b26295-9a04-41fb-8c2d-7633791fd7f9" path="/var/lib/kubelet/pods/e5b26295-9a04-41fb-8c2d-7633791fd7f9/volumes"
Nov 01 23:14:16.179979 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:16.179948    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:14:17.229228 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:17.229197    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"]
Nov 01 23:14:17.229620 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:17.229406    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy" containerID="cri-o://361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b" gracePeriod=30
Nov 01 23:14:18.168061 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168017    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"]
Nov 01 23:14:18.168410 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168397    4837 cpu_manager.go:414] "containerMap: removing container" podUID="e5b26295-9a04-41fb-8c2d-7633791fd7f9" containerName="payload-processing"
Nov 01 23:14:18.168456 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168412    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b26295-9a04-41fb-8c2d-7633791fd7f9" containerName="payload-processing"
Nov 01 23:14:18.168456 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168438    4837 cpu_manager.go:414] "containerMap: removing container" podUID="126de8c6-4972-48a1-8579-8407b9f33750" containerName="revoke-keys"
Nov 01 23:14:18.168456 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168443    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126de8c6-4972-48a1-8579-8407b9f33750" containerName="revoke-keys"
Nov 01 23:14:18.168456 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168453    4837 cpu_manager.go:414] "containerMap: removing container" podUID="0f69b1d7-7b72-4cf9-baf5-f13cebeaa443" containerName="revoke-keys"
Nov 01 23:14:18.168586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168458    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f69b1d7-7b72-4cf9-baf5-f13cebeaa443" containerName="revoke-keys"
Nov 01 23:14:18.168586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168545    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="126de8c6-4972-48a1-8579-8407b9f33750" containerName="revoke-keys"
Nov 01 23:14:18.168586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168555    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f69b1d7-7b72-4cf9-baf5-f13cebeaa443" containerName="revoke-keys"
Nov 01 23:14:18.168586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.168564    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b26295-9a04-41fb-8c2d-7633791fd7f9" containerName="payload-processing"
Nov 01 23:14:18.173114 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.173093    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.226796 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.226770    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"]
Nov 01 23:14:18.231944 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.231920    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:14:18.232337 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.231942    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:14:18.271200 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.271170    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:14:18.271287 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.271186    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:14:18.281014 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.280994    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.281127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.281029    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.281180 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.281171    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gmg\" (UniqueName: \"kubernetes.io/projected/f2aa47b1-c068-4d07-ba91-042ab703717c-kube-api-access-69gmg\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.381567 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.381544    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gmg\" (UniqueName: \"kubernetes.io/projected/f2aa47b1-c068-4d07-ba91-042ab703717c-kube-api-access-69gmg\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.381667 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.381596    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.381667 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.381633    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.382293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.382276    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.383857 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.383839    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.434789 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.434738    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gmg\" (UniqueName: \"kubernetes.io/projected/f2aa47b1-c068-4d07-ba91-042ab703717c-kube-api-access-69gmg\") pod \"maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.486195 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.486177    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:18.814546 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:14:18.814461    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:14:18.850544 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.850482    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"]
Nov 01 23:14:18.853000 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:14:18.852970    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aa47b1_c068_4d07_ba91_042ab703717c.slice/crio-e9cb3fc290641e1d4051fbdbccf3d46ad9e300a13ac3a1654900d75e518231b7 WatchSource:0}: Error finding container e9cb3fc290641e1d4051fbdbccf3d46ad9e300a13ac3a1654900d75e518231b7: Status 404 returned error can't find the container with id e9cb3fc290641e1d4051fbdbccf3d46ad9e300a13ac3a1654900d75e518231b7
Nov 01 23:14:18.999489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:18.999458    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh" event={"ID":"f2aa47b1-c068-4d07-ba91-042ab703717c","Type":"ContainerStarted","Data":"e9cb3fc290641e1d4051fbdbccf3d46ad9e300a13ac3a1654900d75e518231b7"}
Nov 01 23:14:20.005422 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:20.005384    4837 generic.go:356] "Generic (PLEG): container finished" podID="f2aa47b1-c068-4d07-ba91-042ab703717c" containerID="e6ca0d10c3b80171e4938c586f61739cd95ccb3414c3942a9ec7ee64fc7b5fb0" exitCode=0
Nov 01 23:14:20.005805 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:20.005468    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh" event={"ID":"f2aa47b1-c068-4d07-ba91-042ab703717c","Type":"ContainerDied","Data":"e6ca0d10c3b80171e4938c586f61739cd95ccb3414c3942a9ec7ee64fc7b5fb0"}
Nov 01 23:14:21.139873 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.139851    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:21.307913 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.307846    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-openshift-ca\") pod \"f2aa47b1-c068-4d07-ba91-042ab703717c\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") "
Nov 01 23:14:21.307913 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.307899    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2aa47b1-c068-4d07-ba91-042ab703717c-kube-api-access-69gmg\" (UniqueName: \"kubernetes.io/projected/f2aa47b1-c068-4d07-ba91-042ab703717c-kube-api-access-69gmg\") pod \"f2aa47b1-c068-4d07-ba91-042ab703717c\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") "
Nov 01 23:14:21.308157 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.307925    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-xks-ca\") pod \"f2aa47b1-c068-4d07-ba91-042ab703717c\" (UID: \"f2aa47b1-c068-4d07-ba91-042ab703717c\") "
Nov 01 23:14:21.308248 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.308223    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-openshift-ca" pod "f2aa47b1-c068-4d07-ba91-042ab703717c" (UID: "f2aa47b1-c068-4d07-ba91-042ab703717c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:21.309940 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.309917    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-xks-ca" pod "f2aa47b1-c068-4d07-ba91-042ab703717c" (UID: "f2aa47b1-c068-4d07-ba91-042ab703717c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:21.310087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.310069    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2aa47b1-c068-4d07-ba91-042ab703717c-kube-api-access-69gmg" pod "f2aa47b1-c068-4d07-ba91-042ab703717c" (UID: "f2aa47b1-c068-4d07-ba91-042ab703717c"). InnerVolumeSpecName "kube-api-access-69gmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:21.409520 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.409491    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:21.409520 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.409515    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69gmg\" (UniqueName: \"kubernetes.io/projected/f2aa47b1-c068-4d07-ba91-042ab703717c-kube-api-access-69gmg\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:21.409671 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:21.409526    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2aa47b1-c068-4d07-ba91-042ab703717c-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.017226 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.017198    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"
Nov 01 23:14:22.017408 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.017227    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh" event={"ID":"f2aa47b1-c068-4d07-ba91-042ab703717c","Type":"ContainerDied","Data":"e9cb3fc290641e1d4051fbdbccf3d46ad9e300a13ac3a1654900d75e518231b7"}
Nov 01 23:14:22.017408 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.017274    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9cb3fc290641e1d4051fbdbccf3d46ad9e300a13ac3a1654900d75e518231b7"
Nov 01 23:14:22.480069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.480028    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 23:14:22.620179 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620123    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-envoy\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620179 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620154    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-credential-socket\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620184    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-data\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620214    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istiod-ca-cert\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620242    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-socket\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620371    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-token\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-token\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620441    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-kube-api-access-s49tf\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-kube-api-access-s49tf\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620463    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-credential-socket" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:22.620590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620529    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-socket" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:22.620590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620533    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-data" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:22.620787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620593    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-certs\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620634    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-podinfo\") pod \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\" (UID: \"c510ab0f-4027-4c03-aead-96adc4e1cb3a\") "
Nov 01 23:14:22.620787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620660    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istiod-ca-cert" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:22.620953 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620852    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-certs" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:22.621020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.620996    4837 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-data\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.621104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.621015    4837 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istiod-ca-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.621104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.621033    4837 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.621104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.621069    4837 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-workload-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.621104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.621084    4837 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-credential-socket\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.622483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.622456    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-envoy" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:22.622624 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.622607    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-token" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:22.622675 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.622640    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-kube-api-access-s49tf" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "kube-api-access-s49tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:22.622764 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.622746    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-podinfo" pod "c510ab0f-4027-4c03-aead-96adc4e1cb3a" (UID: "c510ab0f-4027-4c03-aead-96adc4e1cb3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:14:22.722288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.722261    4837 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-token\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.722288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.722283    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s49tf\" (UniqueName: \"kubernetes.io/projected/c510ab0f-4027-4c03-aead-96adc4e1cb3a-kube-api-access-s49tf\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.722288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.722292    4837 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-podinfo\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:22.722491 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:22.722302    4837 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c510ab0f-4027-4c03-aead-96adc4e1cb3a-istio-envoy\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:23.023618 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.023587    4837 generic.go:356] "Generic (PLEG): container finished" podID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerID="361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b" exitCode=0
Nov 01 23:14:23.023790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.023638    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"
Nov 01 23:14:23.023790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.023667    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" event={"ID":"c510ab0f-4027-4c03-aead-96adc4e1cb3a","Type":"ContainerDied","Data":"361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b"}
Nov 01 23:14:23.023790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.023726    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg" event={"ID":"c510ab0f-4027-4c03-aead-96adc4e1cb3a","Type":"ContainerDied","Data":"897461cebc48117b203ca20238fa51e44af971b75c561667d649c42da9dab164"}
Nov 01 23:14:23.023790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.023755    4837 scope.go:122] "RemoveContainer" containerID="361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b"
Nov 01 23:14:23.033261 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.033244    4837 scope.go:122] "RemoveContainer" containerID="31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd"
Nov 01 23:14:23.041066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.041034    4837 scope.go:122] "RemoveContainer" containerID="361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b"
Nov 01 23:14:23.041335 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:14:23.041313    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b\": container with ID starting with 361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b not found: ID does not exist" containerID="361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b"
Nov 01 23:14:23.041415 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.041349    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b"} err="failed to get container status \"361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b\": rpc error: code = NotFound desc = could not find container \"361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b\": container with ID starting with 361a85fad69fc8040427abfdb7575cbe4f6c6e52a778507300bfe7496e51692b not found: ID does not exist"
Nov 01 23:14:23.041415 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.041374    4837 scope.go:122] "RemoveContainer" containerID="31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd"
Nov 01 23:14:23.041602 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:14:23.041586    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd\": container with ID starting with 31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd not found: ID does not exist" containerID="31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd"
Nov 01 23:14:23.041654 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.041611    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd"} err="failed to get container status \"31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd\": rpc error: code = NotFound desc = could not find container \"31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd\": container with ID starting with 31114ec1632616ac9cdb0b3043f13813f1eb97f944d153ee8c869668d8fcc3dd not found: ID does not exist"
Nov 01 23:14:23.176121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.176092    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"]
Nov 01 23:14:23.179805 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:23.179783    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-661e9f-openshift-default-79dff869f9-q5gzg"]
Nov 01 23:14:24.523594 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:24.523557    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" path="/var/lib/kubelet/pods/c510ab0f-4027-4c03-aead-96adc4e1cb3a/volumes"
Nov 01 23:14:38.056071 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.056023    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"]
Nov 01 23:14:38.056577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.056429    4837 cpu_manager.go:414] "containerMap: removing container" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy"
Nov 01 23:14:38.056577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.056441    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy"
Nov 01 23:14:38.056577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.056476    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f2aa47b1-c068-4d07-ba91-042ab703717c" containerName="revoke-keys"
Nov 01 23:14:38.056577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.056481    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2aa47b1-c068-4d07-ba91-042ab703717c" containerName="revoke-keys"
Nov 01 23:14:38.056577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.056546    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2aa47b1-c068-4d07-ba91-042ab703717c" containerName="revoke-keys"
Nov 01 23:14:38.056577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.056555    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy"
Nov 01 23:14:38.056577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.056563    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy"
Nov 01 23:14:38.059734 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.059713    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.114164 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.114130    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:14:38.114457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.114430    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:14:38.114600 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.114461    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:14:38.114812 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.114795    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:14:38.117159 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.117126    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"]
Nov 01 23:14:38.155489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.155462    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.155641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.155516    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgmz\" (UniqueName: \"kubernetes.io/projected/045acd3b-71fa-4373-b4f6-f65fa379bc91-kube-api-access-4vgmz\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.155641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.155596    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.257006 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.256976    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.257180 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.257029    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgmz\" (UniqueName: \"kubernetes.io/projected/045acd3b-71fa-4373-b4f6-f65fa379bc91-kube-api-access-4vgmz\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.257233 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.257211    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.257805 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.257781    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.259428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.259409    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.310690 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.310626    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgmz\" (UniqueName: \"kubernetes.io/projected/045acd3b-71fa-4373-b4f6-f65fa379bc91-kube-api-access-4vgmz\") pod \"maas-api-revoke-sub-06513bc9-2mmlp\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.373000 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.372967    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:38.735096 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:38.735069    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"]
Nov 01 23:14:38.736929 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:14:38.736904    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045acd3b_71fa_4373_b4f6_f65fa379bc91.slice/crio-b648026bb80587520dc24a914fe38231c3f256867d6e9acc7d26998b027539f3 WatchSource:0}: Error finding container b648026bb80587520dc24a914fe38231c3f256867d6e9acc7d26998b027539f3: Status 404 returned error can't find the container with id b648026bb80587520dc24a914fe38231c3f256867d6e9acc7d26998b027539f3
Nov 01 23:14:39.101516 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:39.101442    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp" event={"ID":"045acd3b-71fa-4373-b4f6-f65fa379bc91","Type":"ContainerStarted","Data":"f049cda2e85b94723daaf7f8861ddd4abde2464ccbbf2ad12b6067738734f556"}
Nov 01 23:14:39.101861 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:39.101532    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp" event={"ID":"045acd3b-71fa-4373-b4f6-f65fa379bc91","Type":"ContainerStarted","Data":"b648026bb80587520dc24a914fe38231c3f256867d6e9acc7d26998b027539f3"}
Nov 01 23:14:39.208744 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:39.208700    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp" podStartSLOduration=1.2086880070000001 podStartE2EDuration="1.208688007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:14:39.208249638 +0000 UTC m=+3060.797448699" watchObservedRunningTime="2026-11-01 23:14:39.208688007 +0000 UTC m=+3060.797887067"
Nov 01 23:14:40.108250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:40.108217    4837 generic.go:356] "Generic (PLEG): container finished" podID="045acd3b-71fa-4373-b4f6-f65fa379bc91" containerID="f049cda2e85b94723daaf7f8861ddd4abde2464ccbbf2ad12b6067738734f556" exitCode=0
Nov 01 23:14:40.108638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:40.108299    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp" event={"ID":"045acd3b-71fa-4373-b4f6-f65fa379bc91","Type":"ContainerDied","Data":"f049cda2e85b94723daaf7f8861ddd4abde2464ccbbf2ad12b6067738734f556"}
Nov 01 23:14:41.241621 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.241592    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:41.282288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.282265    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-openshift-ca\") pod \"045acd3b-71fa-4373-b4f6-f65fa379bc91\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") "
Nov 01 23:14:41.282424 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.282317    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-xks-ca\") pod \"045acd3b-71fa-4373-b4f6-f65fa379bc91\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") "
Nov 01 23:14:41.282424 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.282341    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045acd3b-71fa-4373-b4f6-f65fa379bc91-kube-api-access-4vgmz\" (UniqueName: \"kubernetes.io/projected/045acd3b-71fa-4373-b4f6-f65fa379bc91-kube-api-access-4vgmz\") pod \"045acd3b-71fa-4373-b4f6-f65fa379bc91\" (UID: \"045acd3b-71fa-4373-b4f6-f65fa379bc91\") "
Nov 01 23:14:41.282614 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.282589    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-openshift-ca" pod "045acd3b-71fa-4373-b4f6-f65fa379bc91" (UID: "045acd3b-71fa-4373-b4f6-f65fa379bc91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:41.284398 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.284374    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-xks-ca" pod "045acd3b-71fa-4373-b4f6-f65fa379bc91" (UID: "045acd3b-71fa-4373-b4f6-f65fa379bc91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:41.284503 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.284447    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045acd3b-71fa-4373-b4f6-f65fa379bc91-kube-api-access-4vgmz" pod "045acd3b-71fa-4373-b4f6-f65fa379bc91" (UID: "045acd3b-71fa-4373-b4f6-f65fa379bc91"). InnerVolumeSpecName "kube-api-access-4vgmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:41.383438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.383376    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:41.383438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.383400    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vgmz\" (UniqueName: \"kubernetes.io/projected/045acd3b-71fa-4373-b4f6-f65fa379bc91-kube-api-access-4vgmz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:41.383438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:41.383412    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045acd3b-71fa-4373-b4f6-f65fa379bc91-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:42.119288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:42.119255    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"
Nov 01 23:14:42.119454 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:42.119289    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp" event={"ID":"045acd3b-71fa-4373-b4f6-f65fa379bc91","Type":"ContainerDied","Data":"b648026bb80587520dc24a914fe38231c3f256867d6e9acc7d26998b027539f3"}
Nov 01 23:14:42.119454 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:42.119336    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b648026bb80587520dc24a914fe38231c3f256867d6e9acc7d26998b027539f3"
Nov 01 23:14:48.814552 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:14:48.814514    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:14:53.778086 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:53.778038    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:14:55.474863 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:55.474825    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"]
Nov 01 23:14:55.519921 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:55.519891    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9dfe9f-k8sgh"]
Nov 01 23:14:56.523208 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:14:56.523175    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2aa47b1-c068-4d07-ba91-042ab703717c" path="/var/lib/kubelet/pods/f2aa47b1-c068-4d07-ba91-042ab703717c/volumes"
Nov 01 23:15:00.161697 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.161661    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"]
Nov 01 23:15:00.162100 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.162073    4837 cpu_manager.go:414] "containerMap: removing container" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy"
Nov 01 23:15:00.162100 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.162086    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c510ab0f-4027-4c03-aead-96adc4e1cb3a" containerName="istio-proxy"
Nov 01 23:15:00.162186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.162119    4837 cpu_manager.go:414] "containerMap: removing container" podUID="045acd3b-71fa-4373-b4f6-f65fa379bc91" containerName="revoke-keys"
Nov 01 23:15:00.162186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.162125    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045acd3b-71fa-4373-b4f6-f65fa379bc91" containerName="revoke-keys"
Nov 01 23:15:00.162259 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.162190    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="045acd3b-71fa-4373-b4f6-f65fa379bc91" containerName="revoke-keys"
Nov 01 23:15:00.166443 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.166424    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:00.219362 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.219334    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"]
Nov 01 23:15:00.250993 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.250962    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jznj\" (UniqueName: \"kubernetes.io/projected/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-kube-api-access-4jznj\") pod \"maas-api-key-cleanup-29892915-7zt9r\" (UID: \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:00.251154 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.251014    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892915-7zt9r\" (UID: \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:00.264006 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.263976    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:15:00.264230 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.264210    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:15:00.264295 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.264257    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:15:00.351779 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.351754    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jznj\" (UniqueName: \"kubernetes.io/projected/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-kube-api-access-4jznj\") pod \"maas-api-key-cleanup-29892915-7zt9r\" (UID: \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:00.351934 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.351822    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892915-7zt9r\" (UID: \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:00.352447 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.352429    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892915-7zt9r\" (UID: \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:00.406153 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.406131    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jznj\" (UniqueName: \"kubernetes.io/projected/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-kube-api-access-4jznj\") pod \"maas-api-key-cleanup-29892915-7zt9r\" (UID: \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:00.478645 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.478620    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:00.844861 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:00.844831    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"]
Nov 01 23:15:00.846819 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:15:00.846779    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af2a9f6_b441_4d4b_a90a_9e3fc0740e19.slice/crio-1b05ea31657c431eb5c1079fb53217568fc122f0d3bdc5fbab033d5608c13300 WatchSource:0}: Error finding container 1b05ea31657c431eb5c1079fb53217568fc122f0d3bdc5fbab033d5608c13300: Status 404 returned error can't find the container with id 1b05ea31657c431eb5c1079fb53217568fc122f0d3bdc5fbab033d5608c13300
Nov 01 23:15:01.208719 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:01.208691    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" event={"ID":"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19","Type":"ContainerStarted","Data":"0258f4c53adc901f921160b420dc2799e9489872d70b2cd2f1d9427c8ab0f1c8"}
Nov 01 23:15:01.209068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:01.208742    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" event={"ID":"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19","Type":"ContainerStarted","Data":"1b05ea31657c431eb5c1079fb53217568fc122f0d3bdc5fbab033d5608c13300"}
Nov 01 23:15:01.315505 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:01.315463    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" podStartSLOduration=1.31544948 podStartE2EDuration="1.31544948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:15:01.315191649 +0000 UTC m=+3082.904390709" watchObservedRunningTime="2026-11-01 23:15:01.31544948 +0000 UTC m=+3082.904648541"
Nov 01 23:15:02.215554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:02.215520    4837 generic.go:356] "Generic (PLEG): container finished" podID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerID="0258f4c53adc901f921160b420dc2799e9489872d70b2cd2f1d9427c8ab0f1c8" exitCode=60
Nov 01 23:15:02.216073 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:02.215609    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" event={"ID":"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19","Type":"ContainerDied","Data":"0258f4c53adc901f921160b420dc2799e9489872d70b2cd2f1d9427c8ab0f1c8"}
Nov 01 23:15:02.216073 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:02.215820    4837 scope.go:122] "RemoveContainer" containerID="0258f4c53adc901f921160b420dc2799e9489872d70b2cd2f1d9427c8ab0f1c8"
Nov 01 23:15:03.221598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:03.221566    4837 generic.go:356] "Generic (PLEG): container finished" podID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerID="f78a6e708ec1d69b4f91adf99ac62e6a659f5632f43bb727c7d0c9935f01d7e1" exitCode=60
Nov 01 23:15:03.221943 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:03.221653    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" event={"ID":"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19","Type":"ContainerDied","Data":"f78a6e708ec1d69b4f91adf99ac62e6a659f5632f43bb727c7d0c9935f01d7e1"}
Nov 01 23:15:03.221943 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:03.221715    4837 scope.go:122] "RemoveContainer" containerID="0258f4c53adc901f921160b420dc2799e9489872d70b2cd2f1d9427c8ab0f1c8"
Nov 01 23:15:03.221943 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:03.221917    4837 scope.go:122] "RemoveContainer" containerID="f78a6e708ec1d69b4f91adf99ac62e6a659f5632f43bb727c7d0c9935f01d7e1"
Nov 01 23:15:03.222201 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:15:03.222183    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892915-7zt9r_odh-ai-gateway-infra(9af2a9f6-b441-4d4b-a90a-9e3fc0740e19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19"
Nov 01 23:15:04.227981 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:04.227956    4837 scope.go:122] "RemoveContainer" containerID="f78a6e708ec1d69b4f91adf99ac62e6a659f5632f43bb727c7d0c9935f01d7e1"
Nov 01 23:15:04.228401 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:15:04.228178    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892915-7zt9r_odh-ai-gateway-infra(9af2a9f6-b441-4d4b-a90a-9e3fc0740e19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19"
Nov 01 23:15:09.980671 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:09.980638    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-8zh64"]
Nov 01 23:15:09.984185 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:09.984168    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.033877 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.033846    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 23:15:10.079867 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.079828    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bkm5p\""
Nov 01 23:15:10.079867 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.079829    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 23:15:10.079867 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.079840    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 23:15:10.082222 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.082200    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:15:10.084188 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.084169    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-8zh64"]
Nov 01 23:15:10.133954 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.133930    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-metrics-certs\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.134112 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.134024    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-webhook-cert\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.134112 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.134096    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nh2\" (UniqueName: \"kubernetes.io/projected/f1310c8c-00da-4353-8f3e-bca3f4bb516e-kube-api-access-q7nh2\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.235480 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.235395    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-metrics-certs\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.235664 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.235480    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-webhook-cert\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.235664 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.235507    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nh2\" (UniqueName: \"kubernetes.io/projected/f1310c8c-00da-4353-8f3e-bca3f4bb516e-kube-api-access-q7nh2\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.238150 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.238119    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-metrics-certs\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.238285 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.238199    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-webhook-cert\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.289871 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.289845    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nh2\" (UniqueName: \"kubernetes.io/projected/f1310c8c-00da-4353-8f3e-bca3f4bb516e-kube-api-access-q7nh2\") pod \"maas-controller-5c9f79b66c-8zh64\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") " pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.296739 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.296712    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:10.659460 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.659437    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-8zh64"]
Nov 01 23:15:10.660821 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:15:10.660789    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1310c8c_00da_4353_8f3e_bca3f4bb516e.slice/crio-53c820e833a224228d40a5e6dc7f1823b23b9dc8e8c9395e3652c373c80c1227 WatchSource:0}: Error finding container 53c820e833a224228d40a5e6dc7f1823b23b9dc8e8c9395e3652c373c80c1227: Status 404 returned error can't find the container with id 53c820e833a224228d40a5e6dc7f1823b23b9dc8e8c9395e3652c373c80c1227
Nov 01 23:15:10.662227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:10.662210    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 23:15:11.261827 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:11.261798    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-8zh64" event={"ID":"f1310c8c-00da-4353-8f3e-bca3f4bb516e","Type":"ContainerStarted","Data":"53c820e833a224228d40a5e6dc7f1823b23b9dc8e8c9395e3652c373c80c1227"}
Nov 01 23:15:13.273858 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:13.273830    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-8zh64" event={"ID":"f1310c8c-00da-4353-8f3e-bca3f4bb516e","Type":"ContainerStarted","Data":"367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2"}
Nov 01 23:15:13.274233 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:13.273933    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:13.425226 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:13.425178    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c9f79b66c-8zh64" podStartSLOduration=1.922949984 podStartE2EDuration="4.425165413s" totalImagesPullingTime="2.502215429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:15:13.424105125 +0000 UTC m=+3095.013304187" watchObservedRunningTime="2026-11-01 23:15:13.425165413 +0000 UTC m=+3095.014364474"
Nov 01 23:15:14.522799 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:14.522773    4837 scope.go:122] "RemoveContainer" containerID="f78a6e708ec1d69b4f91adf99ac62e6a659f5632f43bb727c7d0c9935f01d7e1"
Nov 01 23:15:15.284667 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:15.284639    4837 generic.go:356] "Generic (PLEG): container finished" podID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerID="e545eb02186ce4d4cfa83db3241c7f6a6c6c37ac7507ba1d43a9698418552424" exitCode=60
Nov 01 23:15:15.284792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:15.284712    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" event={"ID":"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19","Type":"ContainerDied","Data":"e545eb02186ce4d4cfa83db3241c7f6a6c6c37ac7507ba1d43a9698418552424"}
Nov 01 23:15:15.284834 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:15.284801    4837 scope.go:122] "RemoveContainer" containerID="f78a6e708ec1d69b4f91adf99ac62e6a659f5632f43bb727c7d0c9935f01d7e1"
Nov 01 23:15:15.285111 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:15.285092    4837 scope.go:122] "RemoveContainer" containerID="e545eb02186ce4d4cfa83db3241c7f6a6c6c37ac7507ba1d43a9698418552424"
Nov 01 23:15:15.285330 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:15:15.285307    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892915-7zt9r_odh-ai-gateway-infra(9af2a9f6-b441-4d4b-a90a-9e3fc0740e19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19"
Nov 01 23:15:16.400748 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:16.400716    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"]
Nov 01 23:15:16.539724 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:16.539703    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:16.692083 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:16.692029    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-kube-api-access-4jznj\" (UniqueName: \"kubernetes.io/projected/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-kube-api-access-4jznj\") pod \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\" (UID: \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\") "
Nov 01 23:15:16.692248 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:16.692127    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-maas-api-service-ca\") pod \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\" (UID: \"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19\") "
Nov 01 23:15:16.692483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:16.692456    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-maas-api-service-ca" pod "9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" (UID: "9af2a9f6-b441-4d4b-a90a-9e3fc0740e19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:15:16.694202 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:16.694173    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-kube-api-access-4jznj" pod "9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" (UID: "9af2a9f6-b441-4d4b-a90a-9e3fc0740e19"). InnerVolumeSpecName "kube-api-access-4jznj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:15:16.792677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:16.792646    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jznj\" (UniqueName: \"kubernetes.io/projected/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-kube-api-access-4jznj\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:16.792798 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:16.792671    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:17.296634 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:17.296607    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"
Nov 01 23:15:17.296634 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:17.296616    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r" event={"ID":"9af2a9f6-b441-4d4b-a90a-9e3fc0740e19","Type":"ContainerDied","Data":"1b05ea31657c431eb5c1079fb53217568fc122f0d3bdc5fbab033d5608c13300"}
Nov 01 23:15:17.296912 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:17.296677    4837 scope.go:122] "RemoveContainer" containerID="e545eb02186ce4d4cfa83db3241c7f6a6c6c37ac7507ba1d43a9698418552424"
Nov 01 23:15:17.448492 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:17.448467    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"]
Nov 01 23:15:17.451019 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:17.450997    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-7zt9r"]
Nov 01 23:15:18.523282 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:18.523246    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" path="/var/lib/kubelet/pods/9af2a9f6-b441-4d4b-a90a-9e3fc0740e19/volumes"
Nov 01 23:15:18.815403 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:15:18.815327    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:15:24.280848 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.280820    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:24.949911 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.949883    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"]
Nov 01 23:15:24.950276 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.950264    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:24.950329 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.950278    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:24.950329 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.950294    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:24.950329 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.950299    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:24.950448 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.950373    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:24.950448 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.950385    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:24.953167 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:24.953149    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.001965 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.001947    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bv4kj\""
Nov 01 23:15:25.052100 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.052075    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"]
Nov 01 23:15:25.064570 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.064542    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.064673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.064579    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.064673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.064598    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gngr\" (UniqueName: \"kubernetes.io/projected/43410c5e-e1c9-4389-b1e0-c12e99842a84-kube-api-access-8gngr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.064751 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.064689    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.064799 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.064758    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.064840 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.064801    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.064908 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.064893    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.064954 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.064920    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.065031 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.065019    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43410c5e-e1c9-4389-b1e0-c12e99842a84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.165616 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.165591    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.165718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.165621    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gngr\" (UniqueName: \"kubernetes.io/projected/43410c5e-e1c9-4389-b1e0-c12e99842a84-kube-api-access-8gngr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.165718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.165656    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.165718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.165690    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.165848 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.165726    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.165848 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.165775    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.165848 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.165803    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.166017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.165876    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43410c5e-e1c9-4389-b1e0-c12e99842a84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.166017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.166007    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.166176 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.166018    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.166176 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.166100    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.166176 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.166156    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.166324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.166296    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.166636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.166619    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43410c5e-e1c9-4389-b1e0-c12e99842a84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.168023 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.168003    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.168181 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.168165    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.219799 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.219733    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43410c5e-e1c9-4389-b1e0-c12e99842a84-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.219905 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.219868    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gngr\" (UniqueName: \"kubernetes.io/projected/43410c5e-e1c9-4389-b1e0-c12e99842a84-kube-api-access-8gngr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zvlcq\" (UID: \"43410c5e-e1c9-4389-b1e0-c12e99842a84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.265782 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.265757    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:25.677832 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.677799    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"]
Nov 01 23:15:25.679510 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:15:25.679477    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43410c5e_e1c9_4389_b1e0_c12e99842a84.slice/crio-3fd77dda2cabc9787426062dbaa94fdf4abf08a05f22a0bb18e38357c268897e WatchSource:0}: Error finding container 3fd77dda2cabc9787426062dbaa94fdf4abf08a05f22a0bb18e38357c268897e: Status 404 returned error can't find the container with id 3fd77dda2cabc9787426062dbaa94fdf4abf08a05f22a0bb18e38357c268897e
Nov 01 23:15:25.681569 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.681539    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:15:25.681645 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.681604    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:15:25.681645 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:25.681635    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:15:26.359725 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:26.359684    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" event={"ID":"43410c5e-e1c9-4389-b1e0-c12e99842a84","Type":"ContainerStarted","Data":"361fc58bc1377e1215ce702cf4b30d9091732931ca16c81e1f75cc415c3b52a4"}
Nov 01 23:15:26.359935 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:26.359773    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" event={"ID":"43410c5e-e1c9-4389-b1e0-c12e99842a84","Type":"ContainerStarted","Data":"3fd77dda2cabc9787426062dbaa94fdf4abf08a05f22a0bb18e38357c268897e"}
Nov 01 23:15:26.563240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:26.563178    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" podStartSLOduration=2.563165686 podStartE2EDuration="2.563165686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:15:26.516338546 +0000 UTC m=+3108.105537607" watchObservedRunningTime="2026-11-01 23:15:26.563165686 +0000 UTC m=+3108.152364749"
Nov 01 23:15:27.266947 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:27.266911    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:27.268543 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:27.268516    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zvlcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 01 23:15:27.268678 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:27.268575    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" podUID="43410c5e-e1c9-4389-b1e0-c12e99842a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 01 23:15:28.266589 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:28.266550    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zvlcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 01 23:15:28.266815 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:28.266607    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" podUID="43410c5e-e1c9-4389-b1e0-c12e99842a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 01 23:15:29.267119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:29.267082    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zvlcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 01 23:15:29.267508 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:29.267144    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" podUID="43410c5e-e1c9-4389-b1e0-c12e99842a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 01 23:15:30.266930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:30.266894    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zvlcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 01 23:15:30.267143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:30.266950    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" podUID="43410c5e-e1c9-4389-b1e0-c12e99842a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 01 23:15:31.266194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:31.266157    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zvlcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 01 23:15:31.266386 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:31.266222    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" podUID="43410c5e-e1c9-4389-b1e0-c12e99842a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 01 23:15:32.266266 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:32.266232    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zvlcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 01 23:15:32.266651 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:32.266293    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq" podUID="43410c5e-e1c9-4389-b1e0-c12e99842a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 01 23:15:34.028223 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.028193    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:34.028734 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.028717    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:34.029746 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.029724    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zvlcq"
Nov 01 23:15:34.058827 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.058796    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"]
Nov 01 23:15:34.059245 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.059228    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:34.059341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.059246    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:34.059413 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.059395    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af2a9f6-b441-4d4b-a90a-9e3fc0740e19" containerName="cleanup"
Nov 01 23:15:34.062643 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.062623    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.111508 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.111489    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:15:34.123701 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.123677    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:15:34.123866 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.123708    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:15:34.123866 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.123756    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:15:34.126095 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.126074    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"]
Nov 01 23:15:34.145102 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.145070    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3caa607-fece-448d-aecd-727b553353aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.145211 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.145114    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvdb\" (UniqueName: \"kubernetes.io/projected/a3caa607-fece-448d-aecd-727b553353aa-kube-api-access-vqvdb\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.145305 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.145287    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3caa607-fece-448d-aecd-727b553353aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.246202 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.246175    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3caa607-fece-448d-aecd-727b553353aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.246359 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.246260    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3caa607-fece-448d-aecd-727b553353aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.246359 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.246289    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvdb\" (UniqueName: \"kubernetes.io/projected/a3caa607-fece-448d-aecd-727b553353aa-kube-api-access-vqvdb\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.247030 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.247002    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3caa607-fece-448d-aecd-727b553353aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.248626 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.248604    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3caa607-fece-448d-aecd-727b553353aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.303964 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.303881    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvdb\" (UniqueName: \"kubernetes.io/projected/a3caa607-fece-448d-aecd-727b553353aa-kube-api-access-vqvdb\") pod \"maas-api-revoke-sub-85237e45-wjbl2\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.375819 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.375791    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:34.740849 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:34.740824    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"]
Nov 01 23:15:34.741982 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:15:34.741950    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3caa607_fece_448d_aecd_727b553353aa.slice/crio-66c46b7f5d96ccef9d872b59ef71cc1a95258b9555251bc0a6b3032390a67e28 WatchSource:0}: Error finding container 66c46b7f5d96ccef9d872b59ef71cc1a95258b9555251bc0a6b3032390a67e28: Status 404 returned error can't find the container with id 66c46b7f5d96ccef9d872b59ef71cc1a95258b9555251bc0a6b3032390a67e28
Nov 01 23:15:35.406079 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:35.405965    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2" event={"ID":"a3caa607-fece-448d-aecd-727b553353aa","Type":"ContainerStarted","Data":"27550a61b66eaeffe7ae3334c258bf7cac86bf9fdc3112eb6230c71b5ef14929"}
Nov 01 23:15:35.406079 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:35.406018    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2" event={"ID":"a3caa607-fece-448d-aecd-727b553353aa","Type":"ContainerStarted","Data":"66c46b7f5d96ccef9d872b59ef71cc1a95258b9555251bc0a6b3032390a67e28"}
Nov 01 23:15:35.513195 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:35.513141    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2" podStartSLOduration=1.513127648 podStartE2EDuration="1.513127648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:15:35.511871342 +0000 UTC m=+3117.101070402" watchObservedRunningTime="2026-11-01 23:15:35.513127648 +0000 UTC m=+3117.102326708"
Nov 01 23:15:36.412503 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:36.412470    4837 generic.go:356] "Generic (PLEG): container finished" podID="a3caa607-fece-448d-aecd-727b553353aa" containerID="27550a61b66eaeffe7ae3334c258bf7cac86bf9fdc3112eb6230c71b5ef14929" exitCode=0
Nov 01 23:15:36.412915 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:36.412534    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2" event={"ID":"a3caa607-fece-448d-aecd-727b553353aa","Type":"ContainerDied","Data":"27550a61b66eaeffe7ae3334c258bf7cac86bf9fdc3112eb6230c71b5ef14929"}
Nov 01 23:15:37.546177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.546152    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:37.573390 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.573364    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3caa607-fece-448d-aecd-727b553353aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3caa607-fece-448d-aecd-727b553353aa-maas-api-openshift-ca\") pod \"a3caa607-fece-448d-aecd-727b553353aa\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") "
Nov 01 23:15:37.573495 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.573406    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3caa607-fece-448d-aecd-727b553353aa-kube-api-access-vqvdb\" (UniqueName: \"kubernetes.io/projected/a3caa607-fece-448d-aecd-727b553353aa-kube-api-access-vqvdb\") pod \"a3caa607-fece-448d-aecd-727b553353aa\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") "
Nov 01 23:15:37.573495 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.573464    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3caa607-fece-448d-aecd-727b553353aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3caa607-fece-448d-aecd-727b553353aa-maas-api-xks-ca\") pod \"a3caa607-fece-448d-aecd-727b553353aa\" (UID: \"a3caa607-fece-448d-aecd-727b553353aa\") "
Nov 01 23:15:37.573706 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.573686    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3caa607-fece-448d-aecd-727b553353aa-maas-api-openshift-ca" pod "a3caa607-fece-448d-aecd-727b553353aa" (UID: "a3caa607-fece-448d-aecd-727b553353aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:15:37.575533 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.575504    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3caa607-fece-448d-aecd-727b553353aa-maas-api-xks-ca" pod "a3caa607-fece-448d-aecd-727b553353aa" (UID: "a3caa607-fece-448d-aecd-727b553353aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:15:37.575652 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.575562    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3caa607-fece-448d-aecd-727b553353aa-kube-api-access-vqvdb" pod "a3caa607-fece-448d-aecd-727b553353aa" (UID: "a3caa607-fece-448d-aecd-727b553353aa"). InnerVolumeSpecName "kube-api-access-vqvdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:15:37.674994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.674937    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3caa607-fece-448d-aecd-727b553353aa-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:37.674994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.674958    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvdb\" (UniqueName: \"kubernetes.io/projected/a3caa607-fece-448d-aecd-727b553353aa-kube-api-access-vqvdb\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:37.674994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.674975    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3caa607-fece-448d-aecd-727b553353aa-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:37.872673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:37.872644    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:15:38.423577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:38.423545    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"
Nov 01 23:15:38.423759 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:38.423566    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2" event={"ID":"a3caa607-fece-448d-aecd-727b553353aa","Type":"ContainerDied","Data":"66c46b7f5d96ccef9d872b59ef71cc1a95258b9555251bc0a6b3032390a67e28"}
Nov 01 23:15:38.423759 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:38.423674    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66c46b7f5d96ccef9d872b59ef71cc1a95258b9555251bc0a6b3032390a67e28"
Nov 01 23:15:46.510059 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.510010    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-8zh64"]
Nov 01 23:15:46.510527 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.510210    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5c9f79b66c-8zh64" podUID="f1310c8c-00da-4353-8f3e-bca3f4bb516e" containerName="manager" containerID="cri-o://367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2" gracePeriod=30
Nov 01 23:15:46.752841 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.752819    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:46.854584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.854529    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-metrics-certs\") pod \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") "
Nov 01 23:15:46.854705 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.854592    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-webhook-cert\") pod \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") "
Nov 01 23:15:46.854705 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.854671    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1310c8c-00da-4353-8f3e-bca3f4bb516e-kube-api-access-q7nh2\" (UniqueName: \"kubernetes.io/projected/f1310c8c-00da-4353-8f3e-bca3f4bb516e-kube-api-access-q7nh2\") pod \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\" (UID: \"f1310c8c-00da-4353-8f3e-bca3f4bb516e\") "
Nov 01 23:15:46.856667 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.856637    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-webhook-cert" pod "f1310c8c-00da-4353-8f3e-bca3f4bb516e" (UID: "f1310c8c-00da-4353-8f3e-bca3f4bb516e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:15:46.856784 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.856698    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1310c8c-00da-4353-8f3e-bca3f4bb516e-kube-api-access-q7nh2" pod "f1310c8c-00da-4353-8f3e-bca3f4bb516e" (UID: "f1310c8c-00da-4353-8f3e-bca3f4bb516e"). InnerVolumeSpecName "kube-api-access-q7nh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:15:46.856784 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.856748    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-metrics-certs" pod "f1310c8c-00da-4353-8f3e-bca3f4bb516e" (UID: "f1310c8c-00da-4353-8f3e-bca3f4bb516e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:15:46.955460 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.955428    4837 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-webhook-cert\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:46.955460 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.955458    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7nh2\" (UniqueName: \"kubernetes.io/projected/f1310c8c-00da-4353-8f3e-bca3f4bb516e-kube-api-access-q7nh2\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:46.955664 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:46.955470    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1310c8c-00da-4353-8f3e-bca3f4bb516e-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:47.464622 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.464588    4837 generic.go:356] "Generic (PLEG): container finished" podID="f1310c8c-00da-4353-8f3e-bca3f4bb516e" containerID="367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2" exitCode=0
Nov 01 23:15:47.464806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.464643    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-8zh64"
Nov 01 23:15:47.464806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.464678    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-8zh64" event={"ID":"f1310c8c-00da-4353-8f3e-bca3f4bb516e","Type":"ContainerDied","Data":"367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2"}
Nov 01 23:15:47.464806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.464733    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-8zh64" event={"ID":"f1310c8c-00da-4353-8f3e-bca3f4bb516e","Type":"ContainerDied","Data":"53c820e833a224228d40a5e6dc7f1823b23b9dc8e8c9395e3652c373c80c1227"}
Nov 01 23:15:47.464806 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.464763    4837 scope.go:122] "RemoveContainer" containerID="367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2"
Nov 01 23:15:47.474444 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.474427    4837 scope.go:122] "RemoveContainer" containerID="367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2"
Nov 01 23:15:47.474729 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:15:47.474712    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2\": container with ID starting with 367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2 not found: ID does not exist" containerID="367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2"
Nov 01 23:15:47.474780 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.474736    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2"} err="failed to get container status \"367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2\": rpc error: code = NotFound desc = could not find container \"367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2\": container with ID starting with 367d74186b9fb3a9106bfba41ab666468748ac4a2c3cd6cc1d586360a1e5edc2 not found: ID does not exist"
Nov 01 23:15:47.617478 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.617450    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-8zh64"]
Nov 01 23:15:47.663461 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:47.663438    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-8zh64"]
Nov 01 23:15:48.522772 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.522735    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1310c8c-00da-4353-8f3e-bca3f4bb516e" path="/var/lib/kubelet/pods/f1310c8c-00da-4353-8f3e-bca3f4bb516e/volumes"
Nov 01 23:15:48.812264 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:15:48.812188    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:15:48.969493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.966054    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-hjmv4"]
Nov 01 23:15:48.969493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.966885    4837 cpu_manager.go:414] "containerMap: removing container" podUID="f1310c8c-00da-4353-8f3e-bca3f4bb516e" containerName="manager"
Nov 01 23:15:48.969493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.966905    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1310c8c-00da-4353-8f3e-bca3f4bb516e" containerName="manager"
Nov 01 23:15:48.969493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.966946    4837 cpu_manager.go:414] "containerMap: removing container" podUID="a3caa607-fece-448d-aecd-727b553353aa" containerName="revoke-keys"
Nov 01 23:15:48.969493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.966956    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3caa607-fece-448d-aecd-727b553353aa" containerName="revoke-keys"
Nov 01 23:15:48.969493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.967164    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1310c8c-00da-4353-8f3e-bca3f4bb516e" containerName="manager"
Nov 01 23:15:48.969493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.967189    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3caa607-fece-448d-aecd-727b553353aa" containerName="revoke-keys"
Nov 01 23:15:48.973360 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:48.973337    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.068588 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.068527    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 23:15:49.068729 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.068625    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 23:15:49.068729 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.068630    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bkm5p\""
Nov 01 23:15:49.068729 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.068720    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 23:15:49.070389 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.070370    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-hjmv4"]
Nov 01 23:15:49.072944 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.072927    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tn9\" (UniqueName: \"kubernetes.io/projected/b8451fe9-c49f-48fe-877c-d86a2a68509c-kube-api-access-96tn9\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.072999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.072978    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8451fe9-c49f-48fe-877c-d86a2a68509c-metrics-certs\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.073161 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.073138    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8451fe9-c49f-48fe-877c-d86a2a68509c-webhook-cert\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.174496 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.174470    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8451fe9-c49f-48fe-877c-d86a2a68509c-metrics-certs\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.174632 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.174519    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8451fe9-c49f-48fe-877c-d86a2a68509c-webhook-cert\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.174632 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.174553    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tn9\" (UniqueName: \"kubernetes.io/projected/b8451fe9-c49f-48fe-877c-d86a2a68509c-kube-api-access-96tn9\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.176970 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.176948    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8451fe9-c49f-48fe-877c-d86a2a68509c-webhook-cert\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.176970 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.176964    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8451fe9-c49f-48fe-877c-d86a2a68509c-metrics-certs\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.228111 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.228088    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tn9\" (UniqueName: \"kubernetes.io/projected/b8451fe9-c49f-48fe-877c-d86a2a68509c-kube-api-access-96tn9\") pod \"maas-controller-5c9f79b66c-hjmv4\" (UID: \"b8451fe9-c49f-48fe-877c-d86a2a68509c\") " pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.286544 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.286524    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:49.647994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:49.647972    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-hjmv4"]
Nov 01 23:15:49.650634 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:15:49.650605    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8451fe9_c49f_48fe_877c_d86a2a68509c.slice/crio-2f05febadcce8d38b3390dd61af6180e8491d9c5300ed49fdcb72d25821d41d8 WatchSource:0}: Error finding container 2f05febadcce8d38b3390dd61af6180e8491d9c5300ed49fdcb72d25821d41d8: Status 404 returned error can't find the container with id 2f05febadcce8d38b3390dd61af6180e8491d9c5300ed49fdcb72d25821d41d8
Nov 01 23:15:50.480760 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:50.480726    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-hjmv4" event={"ID":"b8451fe9-c49f-48fe-877c-d86a2a68509c","Type":"ContainerStarted","Data":"2f05febadcce8d38b3390dd61af6180e8491d9c5300ed49fdcb72d25821d41d8"}
Nov 01 23:15:51.486352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:51.486321    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-hjmv4" event={"ID":"b8451fe9-c49f-48fe-877c-d86a2a68509c","Type":"ContainerStarted","Data":"ff8117aae0a89ad182ba62b0fa2b44d6727c1fa47021fab213d03f65d80191f4"}
Nov 01 23:15:51.486755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:51.486423    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:15:51.596143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:15:51.596093    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c9f79b66c-hjmv4" podStartSLOduration=2.737689995 podStartE2EDuration="3.596076839s" totalImagesPullingTime="858.386844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:15:51.595445476 +0000 UTC m=+3133.184644536" watchObservedRunningTime="2026-11-01 23:15:51.596076839 +0000 UTC m=+3133.185275901"
Nov 01 23:16:02.493308 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:02.493271    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c9f79b66c-hjmv4"
Nov 01 23:16:18.811156 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:16:18.811123    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:16:33.971742 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:33.971707    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:16:42.569019 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.568978    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"]
Nov 01 23:16:42.572705 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.572685    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.621820 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.621797    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:16:42.623011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.622991    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9822fc96-e7f4-4822-b788-81049586e067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.623161 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.623147    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9822fc96-e7f4-4822-b788-81049586e067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.623212 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.623177    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7t8n\" (UniqueName: \"kubernetes.io/projected/9822fc96-e7f4-4822-b788-81049586e067-kube-api-access-l7t8n\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.624233 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.624219    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:16:42.668472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.668444    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:16:42.668590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.668477    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:16:42.670189 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.670170    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"]
Nov 01 23:16:42.724135 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.724115    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9822fc96-e7f4-4822-b788-81049586e067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.724250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.724143    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7t8n\" (UniqueName: \"kubernetes.io/projected/9822fc96-e7f4-4822-b788-81049586e067-kube-api-access-l7t8n\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.724250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.724169    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9822fc96-e7f4-4822-b788-81049586e067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.724948 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.724927    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9822fc96-e7f4-4822-b788-81049586e067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.726627 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.726604    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9822fc96-e7f4-4822-b788-81049586e067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.778141 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.778115    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7t8n\" (UniqueName: \"kubernetes.io/projected/9822fc96-e7f4-4822-b788-81049586e067-kube-api-access-l7t8n\") pod \"maas-api-revoke-sub-c1e271a9-9vt4b\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:42.885332 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:42.885280    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:43.251170 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:43.251146    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"]
Nov 01 23:16:43.252931 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:16:43.252904    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9822fc96_e7f4_4822_b788_81049586e067.slice/crio-3e4e9c8fe45d9fab345c27e289a224a61f85c4cb32e7a60fd19348dd5cae50c7 WatchSource:0}: Error finding container 3e4e9c8fe45d9fab345c27e289a224a61f85c4cb32e7a60fd19348dd5cae50c7: Status 404 returned error can't find the container with id 3e4e9c8fe45d9fab345c27e289a224a61f85c4cb32e7a60fd19348dd5cae50c7
Nov 01 23:16:43.726182 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:43.726150    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b" event={"ID":"9822fc96-e7f4-4822-b788-81049586e067","Type":"ContainerStarted","Data":"de487a46e0f3c2d0d185e70d60e66cd5248e585e685133506812cc205040e87e"}
Nov 01 23:16:43.726528 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:43.726196    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b" event={"ID":"9822fc96-e7f4-4822-b788-81049586e067","Type":"ContainerStarted","Data":"3e4e9c8fe45d9fab345c27e289a224a61f85c4cb32e7a60fd19348dd5cae50c7"}
Nov 01 23:16:43.835352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:43.835280    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b" podStartSLOduration=1.835265589 podStartE2EDuration="1.835265589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:16:43.832953162 +0000 UTC m=+3185.422152225" watchObservedRunningTime="2026-11-01 23:16:43.835265589 +0000 UTC m=+3185.424464650"
Nov 01 23:16:44.732025 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:44.731989    4837 generic.go:356] "Generic (PLEG): container finished" podID="9822fc96-e7f4-4822-b788-81049586e067" containerID="de487a46e0f3c2d0d185e70d60e66cd5248e585e685133506812cc205040e87e" exitCode=0
Nov 01 23:16:44.732420 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:44.732077    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b" event={"ID":"9822fc96-e7f4-4822-b788-81049586e067","Type":"ContainerDied","Data":"de487a46e0f3c2d0d185e70d60e66cd5248e585e685133506812cc205040e87e"}
Nov 01 23:16:45.880332 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:45.880306    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:45.951496 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:45.951465    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9822fc96-e7f4-4822-b788-81049586e067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9822fc96-e7f4-4822-b788-81049586e067-maas-api-xks-ca\") pod \"9822fc96-e7f4-4822-b788-81049586e067\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") "
Nov 01 23:16:45.951641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:45.951512    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9822fc96-e7f4-4822-b788-81049586e067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9822fc96-e7f4-4822-b788-81049586e067-maas-api-openshift-ca\") pod \"9822fc96-e7f4-4822-b788-81049586e067\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") "
Nov 01 23:16:45.951641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:45.951626    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9822fc96-e7f4-4822-b788-81049586e067-kube-api-access-l7t8n\" (UniqueName: \"kubernetes.io/projected/9822fc96-e7f4-4822-b788-81049586e067-kube-api-access-l7t8n\") pod \"9822fc96-e7f4-4822-b788-81049586e067\" (UID: \"9822fc96-e7f4-4822-b788-81049586e067\") "
Nov 01 23:16:45.951881 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:45.951858    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9822fc96-e7f4-4822-b788-81049586e067-maas-api-openshift-ca" pod "9822fc96-e7f4-4822-b788-81049586e067" (UID: "9822fc96-e7f4-4822-b788-81049586e067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:16:45.953615 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:45.953591    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9822fc96-e7f4-4822-b788-81049586e067-maas-api-xks-ca" pod "9822fc96-e7f4-4822-b788-81049586e067" (UID: "9822fc96-e7f4-4822-b788-81049586e067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:16:45.953714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:45.953679    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9822fc96-e7f4-4822-b788-81049586e067-kube-api-access-l7t8n" pod "9822fc96-e7f4-4822-b788-81049586e067" (UID: "9822fc96-e7f4-4822-b788-81049586e067"). InnerVolumeSpecName "kube-api-access-l7t8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:16:46.052639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:46.052583    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7t8n\" (UniqueName: \"kubernetes.io/projected/9822fc96-e7f4-4822-b788-81049586e067-kube-api-access-l7t8n\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:16:46.052639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:46.052605    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9822fc96-e7f4-4822-b788-81049586e067-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:16:46.052639 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:46.052615    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9822fc96-e7f4-4822-b788-81049586e067-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:16:46.742335 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:46.742302    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b" event={"ID":"9822fc96-e7f4-4822-b788-81049586e067","Type":"ContainerDied","Data":"3e4e9c8fe45d9fab345c27e289a224a61f85c4cb32e7a60fd19348dd5cae50c7"}
Nov 01 23:16:46.742515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:46.742397    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4e9c8fe45d9fab345c27e289a224a61f85c4cb32e7a60fd19348dd5cae50c7"
Nov 01 23:16:46.742515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:46.742481    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"
Nov 01 23:16:48.812251 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:16:48.812175    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:16:55.278633 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:16:55.278595    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:17:06.692172 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:06.692133    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:17:10.513192 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.513160    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"]
Nov 01 23:17:10.513601 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.513537    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9822fc96-e7f4-4822-b788-81049586e067" containerName="revoke-keys"
Nov 01 23:17:10.513601 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.513547    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9822fc96-e7f4-4822-b788-81049586e067" containerName="revoke-keys"
Nov 01 23:17:10.513691 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.513622    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9822fc96-e7f4-4822-b788-81049586e067" containerName="revoke-keys"
Nov 01 23:17:10.516664 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.516645    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.616088 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.616053    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"]
Nov 01 23:17:10.650158 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650127    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.650268 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650164    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.650268 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650242    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.650353 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650268    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.650353 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650285    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd9e0f1-2298-4a4b-8946-5327b4787729-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.650353 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650322    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.650453 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650372    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9cw4\" (UniqueName: \"kubernetes.io/projected/1cd9e0f1-2298-4a4b-8946-5327b4787729-kube-api-access-k9cw4\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.650641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650618    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.650679 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.650654    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751363    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751409    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751448    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751478    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751535    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751768 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751565    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751768 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751594    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd9e0f1-2298-4a4b-8946-5327b4787729-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751768 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751633    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751768 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751658    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9cw4\" (UniqueName: \"kubernetes.io/projected/1cd9e0f1-2298-4a4b-8946-5327b4787729-kube-api-access-k9cw4\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751993 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751765    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.751993 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.751784    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.752333 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.752313    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd9e0f1-2298-4a4b-8946-5327b4787729-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.752598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.752575    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.752733 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.752633    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.754209 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.754185    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.754396 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.754377    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.805018 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.804956    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd9e0f1-2298-4a4b-8946-5327b4787729-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.805145 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.805091    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9cw4\" (UniqueName: \"kubernetes.io/projected/1cd9e0f1-2298-4a4b-8946-5327b4787729-kube-api-access-k9cw4\") pod \"maas-default-gateway-openshift-default-7f694df8bc-jkb5h\" (UID: \"1cd9e0f1-2298-4a4b-8946-5327b4787729\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:10.830082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:10.830063    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:11.245774 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:11.245742    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"]
Nov 01 23:17:11.282345 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:17:11.282319    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd9e0f1_2298_4a4b_8946_5327b4787729.slice/crio-a03ed176666ac3d2f8226e8c4516e17e432c52d3173e63e009173150017e49a5 WatchSource:0}: Error finding container a03ed176666ac3d2f8226e8c4516e17e432c52d3173e63e009173150017e49a5: Status 404 returned error can't find the container with id a03ed176666ac3d2f8226e8c4516e17e432c52d3173e63e009173150017e49a5
Nov 01 23:17:11.284303 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:11.284270    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:17:11.284417 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:11.284336    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:17:11.284417 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:11.284369    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:17:11.865183 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:11.865139    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" event={"ID":"1cd9e0f1-2298-4a4b-8946-5327b4787729","Type":"ContainerStarted","Data":"76707982b4bff4367706a4738d39604c3168459a5c68b1f87544295541a93bea"}
Nov 01 23:17:11.865744 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:11.865206    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" event={"ID":"1cd9e0f1-2298-4a4b-8946-5327b4787729","Type":"ContainerStarted","Data":"a03ed176666ac3d2f8226e8c4516e17e432c52d3173e63e009173150017e49a5"}
Nov 01 23:17:11.976341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:11.976285    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" podStartSLOduration=1.9762732889999999 podStartE2EDuration="1.976273289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:17:11.974393319 +0000 UTC m=+3213.563592379" watchObservedRunningTime="2026-11-01 23:17:11.976273289 +0000 UTC m=+3213.565472349"
Nov 01 23:17:12.855077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:12.847582    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:12.855077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:12.849137    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-jkb5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 01 23:17:12.855077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:12.849202    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" podUID="1cd9e0f1-2298-4a4b-8946-5327b4787729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 01 23:17:13.853065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:13.839984    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-jkb5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 01 23:17:13.853065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:13.840101    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" podUID="1cd9e0f1-2298-4a4b-8946-5327b4787729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 01 23:17:14.135121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.131154    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"]
Nov 01 23:17:14.140026 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.140000    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.189864 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.189832    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"]
Nov 01 23:17:14.242070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.239284    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:17:14.242070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.239452    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:17:14.242070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.239484    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:17:14.242070 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.240094    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:17:14.291151 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.291119    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksc9p\" (UniqueName: \"kubernetes.io/projected/85538e02-5dc7-4d77-b3fd-4b2e937db384-kube-api-access-ksc9p\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.291290 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.291178    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.291338 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.291318    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.392620 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.392554    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksc9p\" (UniqueName: \"kubernetes.io/projected/85538e02-5dc7-4d77-b3fd-4b2e937db384-kube-api-access-ksc9p\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.392620 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.392601    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.392803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.392639    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.393361 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.393337    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.394963 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.394945    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.455550 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.455521    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksc9p\" (UniqueName: \"kubernetes.io/projected/85538e02-5dc7-4d77-b3fd-4b2e937db384-kube-api-access-ksc9p\") pod \"maas-api-revoke-sub-2e1f5b5f-9ktd8\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.468881 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.468859    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:14.854341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.831179    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-jkb5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 01 23:17:14.854341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.831231    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" podUID="1cd9e0f1-2298-4a4b-8946-5327b4787729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 01 23:17:14.854341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.831854    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"]
Nov 01 23:17:14.857457 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:17:14.857416    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85538e02_5dc7_4d77_b3fd_4b2e937db384.slice/crio-88877796a9fd7164a1bebd2d2140eb3e37f5e83cf52f1f46bd4bb9a0d5169cef WatchSource:0}: Error finding container 88877796a9fd7164a1bebd2d2140eb3e37f5e83cf52f1f46bd4bb9a0d5169cef: Status 404 returned error can't find the container with id 88877796a9fd7164a1bebd2d2140eb3e37f5e83cf52f1f46bd4bb9a0d5169cef
Nov 01 23:17:14.880817 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:14.880785    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8" event={"ID":"85538e02-5dc7-4d77-b3fd-4b2e937db384","Type":"ContainerStarted","Data":"88877796a9fd7164a1bebd2d2140eb3e37f5e83cf52f1f46bd4bb9a0d5169cef"}
Nov 01 23:17:15.837910 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:15.836122    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-jkb5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 01 23:17:15.837910 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:15.836180    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" podUID="1cd9e0f1-2298-4a4b-8946-5327b4787729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 01 23:17:15.887077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:15.887035    4837 generic.go:356] "Generic (PLEG): container finished" podID="85538e02-5dc7-4d77-b3fd-4b2e937db384" containerID="57bac4b5a90c9c9ca7dc9e5bb8b0ec8bca09d8196d605fff11eb5dc932747ad4" exitCode=0
Nov 01 23:17:15.887451 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:15.887083    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8" event={"ID":"85538e02-5dc7-4d77-b3fd-4b2e937db384","Type":"ContainerDied","Data":"57bac4b5a90c9c9ca7dc9e5bb8b0ec8bca09d8196d605fff11eb5dc932747ad4"}
Nov 01 23:17:16.830928 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:16.830892    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-jkb5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 01 23:17:16.831261 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:16.831229    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" podUID="1cd9e0f1-2298-4a4b-8946-5327b4787729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 01 23:17:17.067097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.067074    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:17.220097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.220068    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-openshift-ca\") pod \"85538e02-5dc7-4d77-b3fd-4b2e937db384\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") "
Nov 01 23:17:17.220279 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.220122    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-xks-ca\") pod \"85538e02-5dc7-4d77-b3fd-4b2e937db384\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") "
Nov 01 23:17:17.220279 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.220217    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85538e02-5dc7-4d77-b3fd-4b2e937db384-kube-api-access-ksc9p\" (UniqueName: \"kubernetes.io/projected/85538e02-5dc7-4d77-b3fd-4b2e937db384-kube-api-access-ksc9p\") pod \"85538e02-5dc7-4d77-b3fd-4b2e937db384\" (UID: \"85538e02-5dc7-4d77-b3fd-4b2e937db384\") "
Nov 01 23:17:17.220419 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.220397    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-openshift-ca" pod "85538e02-5dc7-4d77-b3fd-4b2e937db384" (UID: "85538e02-5dc7-4d77-b3fd-4b2e937db384"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:17:17.220502 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.220479    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:17:17.222232 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.222210    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-xks-ca" pod "85538e02-5dc7-4d77-b3fd-4b2e937db384" (UID: "85538e02-5dc7-4d77-b3fd-4b2e937db384"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:17:17.222335 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.222245    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85538e02-5dc7-4d77-b3fd-4b2e937db384-kube-api-access-ksc9p" pod "85538e02-5dc7-4d77-b3fd-4b2e937db384" (UID: "85538e02-5dc7-4d77-b3fd-4b2e937db384"). InnerVolumeSpecName "kube-api-access-ksc9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:17:17.321492 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.321463    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksc9p\" (UniqueName: \"kubernetes.io/projected/85538e02-5dc7-4d77-b3fd-4b2e937db384-kube-api-access-ksc9p\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:17:17.321492 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.321490    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85538e02-5dc7-4d77-b3fd-4b2e937db384-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:17:17.831104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.831069    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-jkb5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 01 23:17:17.831269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.831124    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h" podUID="1cd9e0f1-2298-4a4b-8946-5327b4787729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 01 23:17:17.899273 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.899245    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8" event={"ID":"85538e02-5dc7-4d77-b3fd-4b2e937db384","Type":"ContainerDied","Data":"88877796a9fd7164a1bebd2d2140eb3e37f5e83cf52f1f46bd4bb9a0d5169cef"}
Nov 01 23:17:17.899432 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.899284    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"
Nov 01 23:17:17.899432 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:17.899291    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88877796a9fd7164a1bebd2d2140eb3e37f5e83cf52f1f46bd4bb9a0d5169cef"
Nov 01 23:17:18.816849 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:17:18.816805    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:17:19.499705 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:19.499672    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:19.499978 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:19.499954    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:19.500873 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:19.500852    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-jkb5h"
Nov 01 23:17:26.534637 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.534608    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"]
Nov 01 23:17:26.535089 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.535059    4837 cpu_manager.go:414] "containerMap: removing container" podUID="85538e02-5dc7-4d77-b3fd-4b2e937db384" containerName="revoke-keys"
Nov 01 23:17:26.535142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.535090    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85538e02-5dc7-4d77-b3fd-4b2e937db384" containerName="revoke-keys"
Nov 01 23:17:26.535186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.535175    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="85538e02-5dc7-4d77-b3fd-4b2e937db384" containerName="revoke-keys"
Nov 01 23:17:26.538617 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.538596    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.590920 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.590889    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:17:26.594004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.593963    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"]
Nov 01 23:17:26.597491 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.597468    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6jp\" (UniqueName: \"kubernetes.io/projected/5591f338-1d2b-4ef4-aa62-3998830f7cae-kube-api-access-xk6jp\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.597631 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.597497    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.597631 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.597525    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.637379 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.637353    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:17:26.637379 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.637376    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:17:26.637574 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.637448    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:17:26.698686 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.698652    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6jp\" (UniqueName: \"kubernetes.io/projected/5591f338-1d2b-4ef4-aa62-3998830f7cae-kube-api-access-xk6jp\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.698686 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.698687    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.698889 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.698716    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.699326 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.699305    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.701108 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.701086    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.753248 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.753211    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6jp\" (UniqueName: \"kubernetes.io/projected/5591f338-1d2b-4ef4-aa62-3998830f7cae-kube-api-access-xk6jp\") pod \"maas-api-revoke-sub-ec843018-txtcf\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:26.844952 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.844879    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:17:26.852389 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:26.852366    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:27.231094 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:27.231064    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"]
Nov 01 23:17:27.232698 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:17:27.232652    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5591f338_1d2b_4ef4_aa62_3998830f7cae.slice/crio-8ada4b64440c979197ec65bfbbf850ffe38c1c1ded2b3d080a39b47112fefaf0 WatchSource:0}: Error finding container 8ada4b64440c979197ec65bfbbf850ffe38c1c1ded2b3d080a39b47112fefaf0: Status 404 returned error can't find the container with id 8ada4b64440c979197ec65bfbbf850ffe38c1c1ded2b3d080a39b47112fefaf0
Nov 01 23:17:27.950011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:27.949972    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf" event={"ID":"5591f338-1d2b-4ef4-aa62-3998830f7cae","Type":"ContainerStarted","Data":"a9ee881b1e2ce776cb9dce363059f938e56c8427bc566c1b2a67af917c8ab5d9"}
Nov 01 23:17:27.950452 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:27.950061    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf" event={"ID":"5591f338-1d2b-4ef4-aa62-3998830f7cae","Type":"ContainerStarted","Data":"8ada4b64440c979197ec65bfbbf850ffe38c1c1ded2b3d080a39b47112fefaf0"}
Nov 01 23:17:28.057602 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:28.057537    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf" podStartSLOduration=2.057524154 podStartE2EDuration="2.057524154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:17:28.056978492 +0000 UTC m=+3229.646177572" watchObservedRunningTime="2026-11-01 23:17:28.057524154 +0000 UTC m=+3229.646723214"
Nov 01 23:17:28.959179 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:28.959141    4837 generic.go:356] "Generic (PLEG): container finished" podID="5591f338-1d2b-4ef4-aa62-3998830f7cae" containerID="a9ee881b1e2ce776cb9dce363059f938e56c8427bc566c1b2a67af917c8ab5d9" exitCode=0
Nov 01 23:17:28.959733 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:28.959220    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf" event={"ID":"5591f338-1d2b-4ef4-aa62-3998830f7cae","Type":"ContainerDied","Data":"a9ee881b1e2ce776cb9dce363059f938e56c8427bc566c1b2a67af917c8ab5d9"}
Nov 01 23:17:30.100267 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.100242    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:30.132279 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.132254    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-openshift-ca\") pod \"5591f338-1d2b-4ef4-aa62-3998830f7cae\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") "
Nov 01 23:17:30.132406 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.132346    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5591f338-1d2b-4ef4-aa62-3998830f7cae-kube-api-access-xk6jp\" (UniqueName: \"kubernetes.io/projected/5591f338-1d2b-4ef4-aa62-3998830f7cae-kube-api-access-xk6jp\") pod \"5591f338-1d2b-4ef4-aa62-3998830f7cae\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") "
Nov 01 23:17:30.132455 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.132412    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-xks-ca\") pod \"5591f338-1d2b-4ef4-aa62-3998830f7cae\" (UID: \"5591f338-1d2b-4ef4-aa62-3998830f7cae\") "
Nov 01 23:17:30.132580 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.132561    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-openshift-ca" pod "5591f338-1d2b-4ef4-aa62-3998830f7cae" (UID: "5591f338-1d2b-4ef4-aa62-3998830f7cae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:17:30.132681 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.132668    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:17:30.134410 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.134388    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-xks-ca" pod "5591f338-1d2b-4ef4-aa62-3998830f7cae" (UID: "5591f338-1d2b-4ef4-aa62-3998830f7cae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:17:30.134540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.134525    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5591f338-1d2b-4ef4-aa62-3998830f7cae-kube-api-access-xk6jp" pod "5591f338-1d2b-4ef4-aa62-3998830f7cae" (UID: "5591f338-1d2b-4ef4-aa62-3998830f7cae"). InnerVolumeSpecName "kube-api-access-xk6jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:17:30.233655 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.233576    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5591f338-1d2b-4ef4-aa62-3998830f7cae-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:17:30.233655 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.233604    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6jp\" (UniqueName: \"kubernetes.io/projected/5591f338-1d2b-4ef4-aa62-3998830f7cae-kube-api-access-xk6jp\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:17:30.970339 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.970306    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"
Nov 01 23:17:30.970522 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.970313    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf" event={"ID":"5591f338-1d2b-4ef4-aa62-3998830f7cae","Type":"ContainerDied","Data":"8ada4b64440c979197ec65bfbbf850ffe38c1c1ded2b3d080a39b47112fefaf0"}
Nov 01 23:17:30.970522 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:30.970428    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ada4b64440c979197ec65bfbbf850ffe38c1c1ded2b3d080a39b47112fefaf0"
Nov 01 23:17:43.572511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:17:43.572481    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:17:48.812025 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:17:48.811981    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:18:01.526695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.526663    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"]
Nov 01 23:18:01.527116 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.527085    4837 cpu_manager.go:414] "containerMap: removing container" podUID="5591f338-1d2b-4ef4-aa62-3998830f7cae" containerName="revoke-keys"
Nov 01 23:18:01.527116 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.527097    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5591f338-1d2b-4ef4-aa62-3998830f7cae" containerName="revoke-keys"
Nov 01 23:18:01.527204 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.527167    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="5591f338-1d2b-4ef4-aa62-3998830f7cae" containerName="revoke-keys"
Nov 01 23:18:01.531646 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.531626    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.583006 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.582970    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:18:01.621311 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.621279    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnmh\" (UniqueName: \"kubernetes.io/projected/d67daa55-b5ea-4ac5-aded-b9f125ffd400-kube-api-access-tlnmh\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.621468 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.621326    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.621468 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.621406    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.629259 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.629234    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:18:01.629259 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.629251    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:18:01.629535 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.629518    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:18:01.631457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.631436    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"]
Nov 01 23:18:01.721919 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.721894    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnmh\" (UniqueName: \"kubernetes.io/projected/d67daa55-b5ea-4ac5-aded-b9f125ffd400-kube-api-access-tlnmh\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.722017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.721923    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.722017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.721940    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.722602 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.722582    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.724133 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.724110    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.775733 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.775716    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnmh\" (UniqueName: \"kubernetes.io/projected/d67daa55-b5ea-4ac5-aded-b9f125ffd400-kube-api-access-tlnmh\") pod \"maas-api-revoke-sub-046e1343-wzlzc\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:01.844676 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:01.844617    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:02.213951 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:02.213923    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"]
Nov 01 23:18:02.216176 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:18:02.216150    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67daa55_b5ea_4ac5_aded_b9f125ffd400.slice/crio-198c39b3ec8cbc95b600fc443011fb172b5d98b7a2eaf2c6e7ae63fd6494d028 WatchSource:0}: Error finding container 198c39b3ec8cbc95b600fc443011fb172b5d98b7a2eaf2c6e7ae63fd6494d028: Status 404 returned error can't find the container with id 198c39b3ec8cbc95b600fc443011fb172b5d98b7a2eaf2c6e7ae63fd6494d028
Nov 01 23:18:03.130397 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:03.130362    4837 generic.go:356] "Generic (PLEG): container finished" podID="d67daa55-b5ea-4ac5-aded-b9f125ffd400" containerID="31d93b755e5132c860e3a12712d3230dae057e5f15799d67687012d75cf4f7d9" exitCode=0
Nov 01 23:18:03.130718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:03.130439    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc" event={"ID":"d67daa55-b5ea-4ac5-aded-b9f125ffd400","Type":"ContainerDied","Data":"31d93b755e5132c860e3a12712d3230dae057e5f15799d67687012d75cf4f7d9"}
Nov 01 23:18:03.130718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:03.130476    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc" event={"ID":"d67daa55-b5ea-4ac5-aded-b9f125ffd400","Type":"ContainerStarted","Data":"198c39b3ec8cbc95b600fc443011fb172b5d98b7a2eaf2c6e7ae63fd6494d028"}
Nov 01 23:18:04.268379 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.268355    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:04.339627 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.339602    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-xks-ca\") pod \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") "
Nov 01 23:18:04.339748 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.339711    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-openshift-ca\") pod \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") "
Nov 01 23:18:04.339748 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.339733    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67daa55-b5ea-4ac5-aded-b9f125ffd400-kube-api-access-tlnmh\" (UniqueName: \"kubernetes.io/projected/d67daa55-b5ea-4ac5-aded-b9f125ffd400-kube-api-access-tlnmh\") pod \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\" (UID: \"d67daa55-b5ea-4ac5-aded-b9f125ffd400\") "
Nov 01 23:18:04.340106 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.340085    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-openshift-ca" pod "d67daa55-b5ea-4ac5-aded-b9f125ffd400" (UID: "d67daa55-b5ea-4ac5-aded-b9f125ffd400"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:18:04.341729 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.341702    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-xks-ca" pod "d67daa55-b5ea-4ac5-aded-b9f125ffd400" (UID: "d67daa55-b5ea-4ac5-aded-b9f125ffd400"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:18:04.341827 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.341789    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67daa55-b5ea-4ac5-aded-b9f125ffd400-kube-api-access-tlnmh" pod "d67daa55-b5ea-4ac5-aded-b9f125ffd400" (UID: "d67daa55-b5ea-4ac5-aded-b9f125ffd400"). InnerVolumeSpecName "kube-api-access-tlnmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:18:04.440917 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.440889    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:04.440917 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.440913    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlnmh\" (UniqueName: \"kubernetes.io/projected/d67daa55-b5ea-4ac5-aded-b9f125ffd400-kube-api-access-tlnmh\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:04.441098 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:04.440923    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67daa55-b5ea-4ac5-aded-b9f125ffd400-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:05.142819 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:05.142793    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"
Nov 01 23:18:05.142995 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:05.142794    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc" event={"ID":"d67daa55-b5ea-4ac5-aded-b9f125ffd400","Type":"ContainerDied","Data":"198c39b3ec8cbc95b600fc443011fb172b5d98b7a2eaf2c6e7ae63fd6494d028"}
Nov 01 23:18:05.142995 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:05.142922    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="198c39b3ec8cbc95b600fc443011fb172b5d98b7a2eaf2c6e7ae63fd6494d028"
Nov 01 23:18:10.983682 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:10.983651    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"]
Nov 01 23:18:10.984116 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:10.984098    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d67daa55-b5ea-4ac5-aded-b9f125ffd400" containerName="revoke-keys"
Nov 01 23:18:10.984168 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:10.984117    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67daa55-b5ea-4ac5-aded-b9f125ffd400" containerName="revoke-keys"
Nov 01 23:18:10.984212 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:10.984204    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67daa55-b5ea-4ac5-aded-b9f125ffd400" containerName="revoke-keys"
Nov 01 23:18:10.987220 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:10.987201    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.030004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.029972    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"]
Nov 01 23:18:11.033346 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.033329    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.084846 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.084816    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"]
Nov 01 23:18:11.093769 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.093744    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.093900 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.093777    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.093900 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.093801    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010f17eb-e378-4b34-9924-55aeba33558a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.093900 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.093818    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.094065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.093937    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.094065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.093962    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.094065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.093992    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvsb\" (UniqueName: \"kubernetes.io/projected/010f17eb-e378-4b34-9924-55aeba33558a-kube-api-access-zcvsb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.094253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094094    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.094253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094194    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.094253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094244    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.094387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094291    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010f17eb-e378-4b34-9924-55aeba33558a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.094387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094318    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.094471 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094388    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.094471 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094426    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxx9\" (UniqueName: \"kubernetes.io/projected/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-kube-api-access-nfxx9\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.094571 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094500    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010f17eb-e378-4b34-9924-55aeba33558a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.094618 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094578    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.094677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094622    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.094677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.094662    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.147538 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.147509    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"]
Nov 01 23:18:11.195755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.195721    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196002 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.195980    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.196186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196160    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvsb\" (UniqueName: \"kubernetes.io/projected/010f17eb-e378-4b34-9924-55aeba33558a-kube-api-access-zcvsb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.196300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196205    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196265    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196301    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196333    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010f17eb-e378-4b34-9924-55aeba33558a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.196489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196364    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196388    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.196489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196417    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxx9\" (UniqueName: \"kubernetes.io/projected/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-kube-api-access-nfxx9\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196447    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010f17eb-e378-4b34-9924-55aeba33558a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.196489 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196479    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196507    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196546    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.196893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196577    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.196893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196615    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.196893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196645    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010f17eb-e378-4b34-9924-55aeba33558a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.196893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.196675    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.197252 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.197072    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.197252 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.197167    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.197430 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.197408    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.197778 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.197758    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.197923 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.197899    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.197994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.197952    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.198230 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.198212    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.198307 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.198283    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.198497 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.198477    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.198945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.198913    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010f17eb-e378-4b34-9924-55aeba33558a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.200025 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.200003    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.200485 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.200464    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010f17eb-e378-4b34-9924-55aeba33558a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.200789 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.200758    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010f17eb-e378-4b34-9924-55aeba33558a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.200876 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.200786    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.250203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.250135    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.250648 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.250631    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010f17eb-e378-4b34-9924-55aeba33558a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.250991 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.250971    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvsb\" (UniqueName: \"kubernetes.io/projected/010f17eb-e378-4b34-9924-55aeba33558a-kube-api-access-zcvsb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pgdbs\" (UID: \"010f17eb-e378-4b34-9924-55aeba33558a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.251132 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.251115    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxx9\" (UniqueName: \"kubernetes.io/projected/cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee-kube-api-access-nfxx9\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pp5tz\" (UID: \"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.299426 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.299398    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:11.346295 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.346263    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:11.718072 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.718026    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"]
Nov 01 23:18:11.753406 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:18:11.753380    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfea7d7_2f9c_41c8_9dbb_b32edf6b67ee.slice/crio-e0dbe144af38d0415ca42edd4658092532af5bf4d6575048f32b0d70d4c4a91c WatchSource:0}: Error finding container e0dbe144af38d0415ca42edd4658092532af5bf4d6575048f32b0d70d4c4a91c: Status 404 returned error can't find the container with id e0dbe144af38d0415ca42edd4658092532af5bf4d6575048f32b0d70d4c4a91c
Nov 01 23:18:11.755485 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.755454    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:18:11.755579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.755521    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:18:11.755579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.755554    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:18:11.767316 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.767295    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"]
Nov 01 23:18:11.771174 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:18:11.770368    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010f17eb_e378_4b34_9924_55aeba33558a.slice/crio-cd564e11240f4e621f300b83395e0e3c25cb4201c55be88bb9a4ad650eb7021a WatchSource:0}: Error finding container cd564e11240f4e621f300b83395e0e3c25cb4201c55be88bb9a4ad650eb7021a: Status 404 returned error can't find the container with id cd564e11240f4e621f300b83395e0e3c25cb4201c55be88bb9a4ad650eb7021a
Nov 01 23:18:11.773376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.773350    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:18:11.773454 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.773412    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:18:11.773454 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:11.773440    4837 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 23:18:12.186716 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.186675    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" event={"ID":"010f17eb-e378-4b34-9924-55aeba33558a","Type":"ContainerStarted","Data":"63210013124e4189c7adc49543c59d0ae11fc3da356d2e064b002216826e018b"}
Nov 01 23:18:12.187223 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.186744    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" event={"ID":"010f17eb-e378-4b34-9924-55aeba33558a","Type":"ContainerStarted","Data":"cd564e11240f4e621f300b83395e0e3c25cb4201c55be88bb9a4ad650eb7021a"}
Nov 01 23:18:12.188556 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.188533    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" event={"ID":"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee","Type":"ContainerStarted","Data":"095770a1a10f77aec8aba83cef45cc71a69ce0a2003cc9025d11e2c6a17f07ef"}
Nov 01 23:18:12.188656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.188581    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" event={"ID":"cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee","Type":"ContainerStarted","Data":"e0dbe144af38d0415ca42edd4658092532af5bf4d6575048f32b0d70d4c4a91c"}
Nov 01 23:18:12.300516 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.300447    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podStartSLOduration=2.300429662 podStartE2EDuration="2.300429662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:12.297990738 +0000 UTC m=+3273.887189804" watchObservedRunningTime="2026-11-01 23:18:12.300429662 +0000 UTC m=+3273.889628723"
Nov 01 23:18:12.302009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.301383    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:12.306258 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.306220    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 01 23:18:12.306438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.306280    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 01 23:18:12.348342 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.348270    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:12.350120 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.350089    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 01 23:18:12.350339 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.350315    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 01 23:18:12.367690 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.367656    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"]
Nov 01 23:18:12.372909 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.372879    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.408378 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.408353    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4f58\" (UniqueName: \"kubernetes.io/projected/14a0daad-4b35-4ea6-929f-73888dcf21f5-kube-api-access-l4f58\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.408610 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.408592    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.408839 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.408824    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.411881 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.411842    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podStartSLOduration=2.411830154 podStartE2EDuration="2.411830154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:12.40938726 +0000 UTC m=+3273.998586320" watchObservedRunningTime="2026-11-01 23:18:12.411830154 +0000 UTC m=+3274.001029268"
Nov 01 23:18:12.423745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.423393    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"]
Nov 01 23:18:12.426562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.426537    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:18:12.426684 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.426668    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:18:12.445127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.442495    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:18:12.445127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.442627    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:18:12.510106 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.510079    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.510253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.510148    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4f58\" (UniqueName: \"kubernetes.io/projected/14a0daad-4b35-4ea6-929f-73888dcf21f5-kube-api-access-l4f58\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.510253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.510172    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.510730 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.510711    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.512961 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.512930    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.583876 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.583839    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4f58\" (UniqueName: \"kubernetes.io/projected/14a0daad-4b35-4ea6-929f-73888dcf21f5-kube-api-access-l4f58\") pod \"maas-api-revoke-sub-d3fba056-zf2bz\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:12.697368 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:12.697269    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:13.109468 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:13.109393    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"]
Nov 01 23:18:13.113330 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:18:13.113305    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a0daad_4b35_4ea6_929f_73888dcf21f5.slice/crio-ea6a47393bc76bc4039fd6a2cf1def627349b3509a327e75a727e775c29df447 WatchSource:0}: Error finding container ea6a47393bc76bc4039fd6a2cf1def627349b3509a327e75a727e775c29df447: Status 404 returned error can't find the container with id ea6a47393bc76bc4039fd6a2cf1def627349b3509a327e75a727e775c29df447
Nov 01 23:18:13.195438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:13.195403    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz" event={"ID":"14a0daad-4b35-4ea6-929f-73888dcf21f5","Type":"ContainerStarted","Data":"ea6a47393bc76bc4039fd6a2cf1def627349b3509a327e75a727e775c29df447"}
Nov 01 23:18:13.300385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:13.300351    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 01 23:18:13.300569 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:13.300416    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 01 23:18:13.348066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:13.347442    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 01 23:18:13.348066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:13.347502    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 01 23:18:14.201825 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:14.201786    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz" event={"ID":"14a0daad-4b35-4ea6-929f-73888dcf21f5","Type":"ContainerStarted","Data":"840dee8ecd6a34301c934c676434b64248ae8838a9d1060026e0a1d9306cde48"}
Nov 01 23:18:14.300760 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:14.300724    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 01 23:18:14.300947 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:14.300774    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 01 23:18:14.311575 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:14.311521    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz" podStartSLOduration=2.311508322 podStartE2EDuration="2.311508322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:14.310285488 +0000 UTC m=+3275.899484549" watchObservedRunningTime="2026-11-01 23:18:14.311508322 +0000 UTC m=+3275.900707404"
Nov 01 23:18:14.350443 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:14.350410    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 01 23:18:14.350762 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:14.350732    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 01 23:18:14.464155 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:14.449662    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:18:15.208637 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:15.208594    4837 generic.go:356] "Generic (PLEG): container finished" podID="14a0daad-4b35-4ea6-929f-73888dcf21f5" containerID="840dee8ecd6a34301c934c676434b64248ae8838a9d1060026e0a1d9306cde48" exitCode=0
Nov 01 23:18:15.209215 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:15.208661    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz" event={"ID":"14a0daad-4b35-4ea6-929f-73888dcf21f5","Type":"ContainerDied","Data":"840dee8ecd6a34301c934c676434b64248ae8838a9d1060026e0a1d9306cde48"}
Nov 01 23:18:15.300495 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:15.300463    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 01 23:18:15.300742 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:15.300718    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 01 23:18:15.347682 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:15.347648    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 01 23:18:15.347995 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:15.347952    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 01 23:18:16.307550 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:16.307072    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 01 23:18:16.307550 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:16.307144    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 01 23:18:16.357586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:16.357457    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 01 23:18:16.357586 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:16.357523    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 01 23:18:17.227305 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.227276    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:17.260840 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.260803    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz" event={"ID":"14a0daad-4b35-4ea6-929f-73888dcf21f5","Type":"ContainerDied","Data":"ea6a47393bc76bc4039fd6a2cf1def627349b3509a327e75a727e775c29df447"}
Nov 01 23:18:17.261188 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.261166    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea6a47393bc76bc4039fd6a2cf1def627349b3509a327e75a727e775c29df447"
Nov 01 23:18:17.265065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.261381    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"
Nov 01 23:18:17.305065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.304566    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 01 23:18:17.305065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.304652    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 01 23:18:17.356066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.353516    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 01 23:18:17.356066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.353572    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 01 23:18:17.431648 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.431598    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-openshift-ca\") pod \"14a0daad-4b35-4ea6-929f-73888dcf21f5\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") "
Nov 01 23:18:17.442276 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.438488    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-openshift-ca" pod "14a0daad-4b35-4ea6-929f-73888dcf21f5" (UID: "14a0daad-4b35-4ea6-929f-73888dcf21f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:18:17.442770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.442746    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a0daad-4b35-4ea6-929f-73888dcf21f5-kube-api-access-l4f58\" (UniqueName: \"kubernetes.io/projected/14a0daad-4b35-4ea6-929f-73888dcf21f5-kube-api-access-l4f58\") pod \"14a0daad-4b35-4ea6-929f-73888dcf21f5\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") "
Nov 01 23:18:17.443162 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.443035    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-xks-ca\") pod \"14a0daad-4b35-4ea6-929f-73888dcf21f5\" (UID: \"14a0daad-4b35-4ea6-929f-73888dcf21f5\") "
Nov 01 23:18:17.454555 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.454521    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:17.469115 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.464489    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-xks-ca" pod "14a0daad-4b35-4ea6-929f-73888dcf21f5" (UID: "14a0daad-4b35-4ea6-929f-73888dcf21f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:18:17.494073 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.493192    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a0daad-4b35-4ea6-929f-73888dcf21f5-kube-api-access-l4f58" pod "14a0daad-4b35-4ea6-929f-73888dcf21f5" (UID: "14a0daad-4b35-4ea6-929f-73888dcf21f5"). InnerVolumeSpecName "kube-api-access-l4f58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:18:17.556915 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.556835    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a0daad-4b35-4ea6-929f-73888dcf21f5-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:17.556915 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:17.556878    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4f58\" (UniqueName: \"kubernetes.io/projected/14a0daad-4b35-4ea6-929f-73888dcf21f5-kube-api-access-l4f58\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:18.300392 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:18.300358    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 01 23:18:18.300564 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:18.300422    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 01 23:18:18.346839 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:18.346804    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 01 23:18:18.346985 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:18.346865    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 01 23:18:18.816428 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:18:18.816386    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:18:19.300455 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:19.300419    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 01 23:18:19.300664 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:19.300472    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 01 23:18:19.347478 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:19.347434    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 01 23:18:19.347702 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:19.347499    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 01 23:18:21.342251 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:21.342209    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:18:21.342790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:21.342277    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:18:21.389234 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:21.389194    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:18:21.389432 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:21.389261    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:18:22.383252 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.383212    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:18:22.383854 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.383819    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:18:22.431218 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.431176    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:18:22.431399 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.431238    4837 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:18:22.703954 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.703912    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:22.704679 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.704648    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:22.705370 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.705353    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs"
Nov 01 23:18:22.739460 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.739438    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:22.739758 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.739724    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:22.760568 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:22.760545    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz"
Nov 01 23:18:24.968205 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:24.968160    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:18:36.859103 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:36.859067    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:18:38.998060 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:38.998020    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:18:38.999804 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:38.999782    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:18:47.328517 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.328476    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"]
Nov 01 23:18:47.329203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.329175    4837 cpu_manager.go:414] "containerMap: removing container" podUID="14a0daad-4b35-4ea6-929f-73888dcf21f5" containerName="revoke-keys"
Nov 01 23:18:47.329203 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.329202    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a0daad-4b35-4ea6-929f-73888dcf21f5" containerName="revoke-keys"
Nov 01 23:18:47.329429 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.329313    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a0daad-4b35-4ea6-929f-73888dcf21f5" containerName="revoke-keys"
Nov 01 23:18:47.332988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.332965    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.382282 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.382255    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:18:47.382429 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.382321    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:18:47.428928 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.428900    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:18:47.429114 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.429083    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:18:47.431244 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.431225    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"]
Nov 01 23:18:47.449211 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.449180    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.449326 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.449244    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlzd\" (UniqueName: \"kubernetes.io/projected/448ab2fc-384d-476b-a321-24463fda13c9-kube-api-access-wdlzd\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.449372 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.449328    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.549945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.549923    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.550091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.549965    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlzd\" (UniqueName: \"kubernetes.io/projected/448ab2fc-384d-476b-a321-24463fda13c9-kube-api-access-wdlzd\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.550091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.549991    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.550578 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.550559    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.552283 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.552264    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.603636 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.603584    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlzd\" (UniqueName: \"kubernetes.io/projected/448ab2fc-384d-476b-a321-24463fda13c9-kube-api-access-wdlzd\") pod \"maas-api-revoke-sub-ae594979-9l8pn\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:47.645948 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:47.645909    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:48.040356 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:48.040328    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"]
Nov 01 23:18:48.041337 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:18:48.041306    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448ab2fc_384d_476b_a321_24463fda13c9.slice/crio-1135ef2356d1470e1fb5783b090dee90244b530982550de4db13681b10fe0253 WatchSource:0}: Error finding container 1135ef2356d1470e1fb5783b090dee90244b530982550de4db13681b10fe0253: Status 404 returned error can't find the container with id 1135ef2356d1470e1fb5783b090dee90244b530982550de4db13681b10fe0253
Nov 01 23:18:48.448674 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:48.448650    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn" event={"ID":"448ab2fc-384d-476b-a321-24463fda13c9","Type":"ContainerStarted","Data":"3bacdc76885f21b199119e6623bf47baf8a754783ffe93ec941bd831e7f2c1bc"}
Nov 01 23:18:48.449026 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:48.448697    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn" event={"ID":"448ab2fc-384d-476b-a321-24463fda13c9","Type":"ContainerStarted","Data":"1135ef2356d1470e1fb5783b090dee90244b530982550de4db13681b10fe0253"}
Nov 01 23:18:48.556022 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:48.555973    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn" podStartSLOduration=1.555956227 podStartE2EDuration="1.555956227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:48.55528664 +0000 UTC m=+3310.144485703" watchObservedRunningTime="2026-11-01 23:18:48.555956227 +0000 UTC m=+3310.145155289"
Nov 01 23:18:48.811880 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:18:48.811845    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:18:49.455619 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:49.455589    4837 generic.go:356] "Generic (PLEG): container finished" podID="448ab2fc-384d-476b-a321-24463fda13c9" containerID="3bacdc76885f21b199119e6623bf47baf8a754783ffe93ec941bd831e7f2c1bc" exitCode=0
Nov 01 23:18:49.455982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:49.455646    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn" event={"ID":"448ab2fc-384d-476b-a321-24463fda13c9","Type":"ContainerDied","Data":"3bacdc76885f21b199119e6623bf47baf8a754783ffe93ec941bd831e7f2c1bc"}
Nov 01 23:18:50.604546 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.604522    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:50.678651 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.678628    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-xks-ca\") pod \"448ab2fc-384d-476b-a321-24463fda13c9\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") "
Nov 01 23:18:50.678787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.678696    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448ab2fc-384d-476b-a321-24463fda13c9-kube-api-access-wdlzd\" (UniqueName: \"kubernetes.io/projected/448ab2fc-384d-476b-a321-24463fda13c9-kube-api-access-wdlzd\") pod \"448ab2fc-384d-476b-a321-24463fda13c9\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") "
Nov 01 23:18:50.678787 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.678736    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-openshift-ca\") pod \"448ab2fc-384d-476b-a321-24463fda13c9\" (UID: \"448ab2fc-384d-476b-a321-24463fda13c9\") "
Nov 01 23:18:50.679159 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.679132    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-openshift-ca" pod "448ab2fc-384d-476b-a321-24463fda13c9" (UID: "448ab2fc-384d-476b-a321-24463fda13c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:18:50.680675 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.680651    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448ab2fc-384d-476b-a321-24463fda13c9-kube-api-access-wdlzd" pod "448ab2fc-384d-476b-a321-24463fda13c9" (UID: "448ab2fc-384d-476b-a321-24463fda13c9"). InnerVolumeSpecName "kube-api-access-wdlzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:18:50.680765 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.680670    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-xks-ca" pod "448ab2fc-384d-476b-a321-24463fda13c9" (UID: "448ab2fc-384d-476b-a321-24463fda13c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:18:50.779559 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.779496    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdlzd\" (UniqueName: \"kubernetes.io/projected/448ab2fc-384d-476b-a321-24463fda13c9-kube-api-access-wdlzd\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:50.779559 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.779525    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:50.779559 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:50.779539    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448ab2fc-384d-476b-a321-24463fda13c9-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:18:51.466928 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:51.466898    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"
Nov 01 23:18:51.467126 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:51.466937    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn" event={"ID":"448ab2fc-384d-476b-a321-24463fda13c9","Type":"ContainerDied","Data":"1135ef2356d1470e1fb5783b090dee90244b530982550de4db13681b10fe0253"}
Nov 01 23:18:51.467126 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:51.466992    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1135ef2356d1470e1fb5783b090dee90244b530982550de4db13681b10fe0253"
Nov 01 23:18:59.972669 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:18:59.972632    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:19:00.042910 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:00.042873    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"]
Nov 01 23:19:00.089507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:00.089474    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5b804f5-j6z9n"]
Nov 01 23:19:00.523321 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:00.523292    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126de8c6-4972-48a1-8579-8407b9f33750" path="/var/lib/kubelet/pods/126de8c6-4972-48a1-8579-8407b9f33750/volumes"
Nov 01 23:19:09.089276 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:09.089236    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:19:15.315817 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.315783    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"]
Nov 01 23:19:15.316273 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.316258    4837 cpu_manager.go:414] "containerMap: removing container" podUID="448ab2fc-384d-476b-a321-24463fda13c9" containerName="revoke-keys"
Nov 01 23:19:15.316324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.316274    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448ab2fc-384d-476b-a321-24463fda13c9" containerName="revoke-keys"
Nov 01 23:19:15.316363 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.316355    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="448ab2fc-384d-476b-a321-24463fda13c9" containerName="revoke-keys"
Nov 01 23:19:15.319465 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.319446    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.368445 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.368423    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:19:15.415140 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.415110    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:19:15.415140 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.415132    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:19:15.417389 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.417368    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"]
Nov 01 23:19:15.451248 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.451220    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.451376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.451263    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.451429 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.451378    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccp88\" (UniqueName: \"kubernetes.io/projected/28372765-75c6-4a89-85cf-d8f63a40fc2e-kube-api-access-ccp88\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.470097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.470071    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:19:15.552235 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.552209    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccp88\" (UniqueName: \"kubernetes.io/projected/28372765-75c6-4a89-85cf-d8f63a40fc2e-kube-api-access-ccp88\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.552356 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.552309    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.552356 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.552344    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.552948 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.552927    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.554627 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.554608    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.605745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.605681    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccp88\" (UniqueName: \"kubernetes.io/projected/28372765-75c6-4a89-85cf-d8f63a40fc2e-kube-api-access-ccp88\") pod \"maas-api-revoke-sub-a4aef7d7-4vhr4\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.631982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.631954    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:15.994976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:15.994950    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"]
Nov 01 23:19:15.996966 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:19:15.996939    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28372765_75c6_4a89_85cf_d8f63a40fc2e.slice/crio-0b05af8057c7a8e2be6092e54b8155d54ec34fc8aaf7b5b133f54ba6d5947593 WatchSource:0}: Error finding container 0b05af8057c7a8e2be6092e54b8155d54ec34fc8aaf7b5b133f54ba6d5947593: Status 404 returned error can't find the container with id 0b05af8057c7a8e2be6092e54b8155d54ec34fc8aaf7b5b133f54ba6d5947593
Nov 01 23:19:16.640113 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:16.640077    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4" event={"ID":"28372765-75c6-4a89-85cf-d8f63a40fc2e","Type":"ContainerStarted","Data":"ea1ddf344b07c6ced34f01774b4cad8cad1dc4c8f16ceb0d6cd65a8537c8d53a"}
Nov 01 23:19:16.640485 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:16.640135    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4" event={"ID":"28372765-75c6-4a89-85cf-d8f63a40fc2e","Type":"ContainerStarted","Data":"0b05af8057c7a8e2be6092e54b8155d54ec34fc8aaf7b5b133f54ba6d5947593"}
Nov 01 23:19:16.747250 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:16.747201    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4" podStartSLOduration=1.747187567 podStartE2EDuration="1.747187567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:19:16.745435788 +0000 UTC m=+3338.334634849" watchObservedRunningTime="2026-11-01 23:19:16.747187567 +0000 UTC m=+3338.336386628"
Nov 01 23:19:17.646266 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:17.646232    4837 generic.go:356] "Generic (PLEG): container finished" podID="28372765-75c6-4a89-85cf-d8f63a40fc2e" containerID="ea1ddf344b07c6ced34f01774b4cad8cad1dc4c8f16ceb0d6cd65a8537c8d53a" exitCode=0
Nov 01 23:19:17.646627 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:17.646318    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4" event={"ID":"28372765-75c6-4a89-85cf-d8f63a40fc2e","Type":"ContainerDied","Data":"ea1ddf344b07c6ced34f01774b4cad8cad1dc4c8f16ceb0d6cd65a8537c8d53a"}
Nov 01 23:19:18.789572 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.789551    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:18.811274 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:19:18.811245    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:19:18.883677 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.883655    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28372765-75c6-4a89-85cf-d8f63a40fc2e-kube-api-access-ccp88\" (UniqueName: \"kubernetes.io/projected/28372765-75c6-4a89-85cf-d8f63a40fc2e-kube-api-access-ccp88\") pod \"28372765-75c6-4a89-85cf-d8f63a40fc2e\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") "
Nov 01 23:19:18.883821 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.883811    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-xks-ca\") pod \"28372765-75c6-4a89-85cf-d8f63a40fc2e\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") "
Nov 01 23:19:18.883870 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.883846    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-openshift-ca\") pod \"28372765-75c6-4a89-85cf-d8f63a40fc2e\" (UID: \"28372765-75c6-4a89-85cf-d8f63a40fc2e\") "
Nov 01 23:19:18.884239 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.884217    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-openshift-ca" pod "28372765-75c6-4a89-85cf-d8f63a40fc2e" (UID: "28372765-75c6-4a89-85cf-d8f63a40fc2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:19:18.885617 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.885595    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28372765-75c6-4a89-85cf-d8f63a40fc2e-kube-api-access-ccp88" pod "28372765-75c6-4a89-85cf-d8f63a40fc2e" (UID: "28372765-75c6-4a89-85cf-d8f63a40fc2e"). InnerVolumeSpecName "kube-api-access-ccp88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:19:18.885897 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.885877    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-xks-ca" pod "28372765-75c6-4a89-85cf-d8f63a40fc2e" (UID: "28372765-75c6-4a89-85cf-d8f63a40fc2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:19:18.984857 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.984830    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccp88\" (UniqueName: \"kubernetes.io/projected/28372765-75c6-4a89-85cf-d8f63a40fc2e-kube-api-access-ccp88\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:19:18.984857 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.984855    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:19:18.985028 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:18.984865    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28372765-75c6-4a89-85cf-d8f63a40fc2e-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:19:19.658967 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:19.658942    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"
Nov 01 23:19:19.659156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:19.658971    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4" event={"ID":"28372765-75c6-4a89-85cf-d8f63a40fc2e","Type":"ContainerDied","Data":"0b05af8057c7a8e2be6092e54b8155d54ec34fc8aaf7b5b133f54ba6d5947593"}
Nov 01 23:19:19.659156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:19.659021    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b05af8057c7a8e2be6092e54b8155d54ec34fc8aaf7b5b133f54ba6d5947593"
Nov 01 23:19:28.080967 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:28.080936    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:19:39.880878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:39.880843    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:19:40.279020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:40.278991    4837 scope.go:122] "RemoveContainer" containerID="80195ddcc9f463b84817ec394179381cf2c6ca627bdf7320b91e0662d4b3abcd"
Nov 01 23:19:40.287590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:40.287568    4837 scope.go:122] "RemoveContainer" containerID="1fd8eded7261a499ba65abb8fc1c9ecea05f20cbde9960781fd08c30459d21ef"
Nov 01 23:19:42.048706 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:42.048671    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"]
Nov 01 23:19:42.094937 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:42.094890    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06513bc9-2mmlp"]
Nov 01 23:19:42.524932 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:42.524891    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045acd3b-71fa-4373-b4f6-f65fa379bc91" path="/var/lib/kubelet/pods/045acd3b-71fa-4373-b4f6-f65fa379bc91/volumes"
Nov 01 23:19:45.964463 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:45.964435    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"]
Nov 01 23:19:45.964882 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:45.964842    4837 cpu_manager.go:414] "containerMap: removing container" podUID="28372765-75c6-4a89-85cf-d8f63a40fc2e" containerName="revoke-keys"
Nov 01 23:19:45.964882 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:45.964855    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28372765-75c6-4a89-85cf-d8f63a40fc2e" containerName="revoke-keys"
Nov 01 23:19:45.964967 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:45.964955    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="28372765-75c6-4a89-85cf-d8f63a40fc2e" containerName="revoke-keys"
Nov 01 23:19:45.969385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:45.969364    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.007112 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.007086    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.007245 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.007121    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.007245 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.007153    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nqk8\" (UniqueName: \"kubernetes.io/projected/8dabc6f2-fd11-4ac5-90fc-410372242a08-kube-api-access-8nqk8\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.020947 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.020922    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:19:46.023478 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.023446    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"]
Nov 01 23:19:46.067503 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.067474    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:19:46.067503 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.067477    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:19:46.067699 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.067617    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:19:46.108127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.108097    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nqk8\" (UniqueName: \"kubernetes.io/projected/8dabc6f2-fd11-4ac5-90fc-410372242a08-kube-api-access-8nqk8\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.108265 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.108204    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.108265 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.108229    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.108938 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.108920    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.110637 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.110618    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.162059 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.162023    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nqk8\" (UniqueName: \"kubernetes.io/projected/8dabc6f2-fd11-4ac5-90fc-410372242a08-kube-api-access-8nqk8\") pod \"maas-api-revoke-sub-79805ca9-n99mz\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.282069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.281987    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:46.645022 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.644995    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"]
Nov 01 23:19:46.646963 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:19:46.646936    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dabc6f2_fd11_4ac5_90fc_410372242a08.slice/crio-b23645f5cef89158065532fa28718fc5492c7150e61b4cc2371ee20947ef3a2c WatchSource:0}: Error finding container b23645f5cef89158065532fa28718fc5492c7150e61b4cc2371ee20947ef3a2c: Status 404 returned error can't find the container with id b23645f5cef89158065532fa28718fc5492c7150e61b4cc2371ee20947ef3a2c
Nov 01 23:19:46.840747 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:46.840719    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz" event={"ID":"8dabc6f2-fd11-4ac5-90fc-410372242a08","Type":"ContainerStarted","Data":"b23645f5cef89158065532fa28718fc5492c7150e61b4cc2371ee20947ef3a2c"}
Nov 01 23:19:47.847011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:47.846938    4837 generic.go:356] "Generic (PLEG): container finished" podID="8dabc6f2-fd11-4ac5-90fc-410372242a08" containerID="736cb062350566a876c36d2a88eaad4f16efdbe9efc1ca1279b05a1100282414" exitCode=0
Nov 01 23:19:47.847394 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:47.847019    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz" event={"ID":"8dabc6f2-fd11-4ac5-90fc-410372242a08","Type":"ContainerDied","Data":"736cb062350566a876c36d2a88eaad4f16efdbe9efc1ca1279b05a1100282414"}
Nov 01 23:19:48.812032 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:19:48.811999    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:19:48.990177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:48.990154    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:49.034695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.034672    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-xks-ca\") pod \"8dabc6f2-fd11-4ac5-90fc-410372242a08\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") "
Nov 01 23:19:49.034827 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.034795    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dabc6f2-fd11-4ac5-90fc-410372242a08-kube-api-access-8nqk8\" (UniqueName: \"kubernetes.io/projected/8dabc6f2-fd11-4ac5-90fc-410372242a08-kube-api-access-8nqk8\") pod \"8dabc6f2-fd11-4ac5-90fc-410372242a08\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") "
Nov 01 23:19:49.034878 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.034830    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-openshift-ca\") pod \"8dabc6f2-fd11-4ac5-90fc-410372242a08\" (UID: \"8dabc6f2-fd11-4ac5-90fc-410372242a08\") "
Nov 01 23:19:49.035212 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.035191    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-openshift-ca" pod "8dabc6f2-fd11-4ac5-90fc-410372242a08" (UID: "8dabc6f2-fd11-4ac5-90fc-410372242a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:19:49.036705 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.036685    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-xks-ca" pod "8dabc6f2-fd11-4ac5-90fc-410372242a08" (UID: "8dabc6f2-fd11-4ac5-90fc-410372242a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:19:49.036790 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.036755    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dabc6f2-fd11-4ac5-90fc-410372242a08-kube-api-access-8nqk8" pod "8dabc6f2-fd11-4ac5-90fc-410372242a08" (UID: "8dabc6f2-fd11-4ac5-90fc-410372242a08"). InnerVolumeSpecName "kube-api-access-8nqk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:19:49.136327 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.136268    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nqk8\" (UniqueName: \"kubernetes.io/projected/8dabc6f2-fd11-4ac5-90fc-410372242a08-kube-api-access-8nqk8\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:19:49.136327 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.136292    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:19:49.136327 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.136302    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabc6f2-fd11-4ac5-90fc-410372242a08-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:19:49.858282 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.858257    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"
Nov 01 23:19:49.858561 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.858254    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz" event={"ID":"8dabc6f2-fd11-4ac5-90fc-410372242a08","Type":"ContainerDied","Data":"b23645f5cef89158065532fa28718fc5492c7150e61b4cc2371ee20947ef3a2c"}
Nov 01 23:19:49.858561 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:19:49.858379    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23645f5cef89158065532fa28718fc5492c7150e61b4cc2371ee20947ef3a2c"
Nov 01 23:20:00.355999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:00.355962    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:20:09.777901 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.777867    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"]
Nov 01 23:20:09.778317 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.778302    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8dabc6f2-fd11-4ac5-90fc-410372242a08" containerName="revoke-keys"
Nov 01 23:20:09.778364 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.778318    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dabc6f2-fd11-4ac5-90fc-410372242a08" containerName="revoke-keys"
Nov 01 23:20:09.778446 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.778435    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dabc6f2-fd11-4ac5-90fc-410372242a08" containerName="revoke-keys"
Nov 01 23:20:09.781603 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.781579    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:09.830657 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.830628    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:20:09.830862 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.830727    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:20:09.866075 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.866025    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:09.866253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.866224    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbb5p\" (UniqueName: \"kubernetes.io/projected/b4d71455-c93a-4c15-bd57-d46e600703fc-kube-api-access-zbb5p\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:09.866331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.866302    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:09.876995 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.876955    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:20:09.877155 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.876999    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:20:09.879295 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.879275    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"]
Nov 01 23:20:09.967728 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.967696    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbb5p\" (UniqueName: \"kubernetes.io/projected/b4d71455-c93a-4c15-bd57-d46e600703fc-kube-api-access-zbb5p\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:09.967929 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.967749    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:09.967929 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.967797    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:09.968453 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.968434    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:09.970089 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:09.970072    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:10.021932 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:10.021905    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbb5p\" (UniqueName: \"kubernetes.io/projected/b4d71455-c93a-4c15-bd57-d46e600703fc-kube-api-access-zbb5p\") pod \"maas-api-revoke-sub-0b7e8edb-xzdcg\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:10.093860 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:10.093805    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:10.455789 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:10.455760    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"]
Nov 01 23:20:10.457359 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:20:10.457325    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d71455_c93a_4c15_bd57_d46e600703fc.slice/crio-a1bb6e7d203e3017ae6a592c2b8e7a1223f4776f80403c9152fb10fbacb6da6e WatchSource:0}: Error finding container a1bb6e7d203e3017ae6a592c2b8e7a1223f4776f80403c9152fb10fbacb6da6e: Status 404 returned error can't find the container with id a1bb6e7d203e3017ae6a592c2b8e7a1223f4776f80403c9152fb10fbacb6da6e
Nov 01 23:20:10.999632 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:10.999598    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg" event={"ID":"b4d71455-c93a-4c15-bd57-d46e600703fc","Type":"ContainerStarted","Data":"344590d2818547cbd285a09249b9b5fc7328d4899c2c3b9c86df811a1f57ac01"}
Nov 01 23:20:11.000018 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:10.999650    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg" event={"ID":"b4d71455-c93a-4c15-bd57-d46e600703fc","Type":"ContainerStarted","Data":"a1bb6e7d203e3017ae6a592c2b8e7a1223f4776f80403c9152fb10fbacb6da6e"}
Nov 01 23:20:11.107983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:11.107939    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg" podStartSLOduration=2.107925933 podStartE2EDuration="2.107925933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:20:11.106955192 +0000 UTC m=+3392.696154252" watchObservedRunningTime="2026-11-01 23:20:11.107925933 +0000 UTC m=+3392.697124993"
Nov 01 23:20:12.007795 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:12.007760    4837 generic.go:356] "Generic (PLEG): container finished" podID="b4d71455-c93a-4c15-bd57-d46e600703fc" containerID="344590d2818547cbd285a09249b9b5fc7328d4899c2c3b9c86df811a1f57ac01" exitCode=0
Nov 01 23:20:12.008221 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:12.007854    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg" event={"ID":"b4d71455-c93a-4c15-bd57-d46e600703fc","Type":"ContainerDied","Data":"344590d2818547cbd285a09249b9b5fc7328d4899c2c3b9c86df811a1f57ac01"}
Nov 01 23:20:13.146562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.146536    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:13.299351 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.299284    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-openshift-ca\") pod \"b4d71455-c93a-4c15-bd57-d46e600703fc\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") "
Nov 01 23:20:13.299496 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.299365    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-xks-ca\") pod \"b4d71455-c93a-4c15-bd57-d46e600703fc\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") "
Nov 01 23:20:13.299496 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.299484    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d71455-c93a-4c15-bd57-d46e600703fc-kube-api-access-zbb5p\" (UniqueName: \"kubernetes.io/projected/b4d71455-c93a-4c15-bd57-d46e600703fc-kube-api-access-zbb5p\") pod \"b4d71455-c93a-4c15-bd57-d46e600703fc\" (UID: \"b4d71455-c93a-4c15-bd57-d46e600703fc\") "
Nov 01 23:20:13.299637 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.299615    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-openshift-ca" pod "b4d71455-c93a-4c15-bd57-d46e600703fc" (UID: "b4d71455-c93a-4c15-bd57-d46e600703fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:20:13.299827 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.299810    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:13.301402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.301383    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-xks-ca" pod "b4d71455-c93a-4c15-bd57-d46e600703fc" (UID: "b4d71455-c93a-4c15-bd57-d46e600703fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:20:13.301515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.301499    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d71455-c93a-4c15-bd57-d46e600703fc-kube-api-access-zbb5p" pod "b4d71455-c93a-4c15-bd57-d46e600703fc" (UID: "b4d71455-c93a-4c15-bd57-d46e600703fc"). InnerVolumeSpecName "kube-api-access-zbb5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:20:13.400403 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.400375    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d71455-c93a-4c15-bd57-d46e600703fc-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:13.400403 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:13.400399    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbb5p\" (UniqueName: \"kubernetes.io/projected/b4d71455-c93a-4c15-bd57-d46e600703fc-kube-api-access-zbb5p\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:14.019475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:14.019450    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"
Nov 01 23:20:14.019650 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:14.019490    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg" event={"ID":"b4d71455-c93a-4c15-bd57-d46e600703fc","Type":"ContainerDied","Data":"a1bb6e7d203e3017ae6a592c2b8e7a1223f4776f80403c9152fb10fbacb6da6e"}
Nov 01 23:20:14.019650 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:14.019535    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1bb6e7d203e3017ae6a592c2b8e7a1223f4776f80403c9152fb10fbacb6da6e"
Nov 01 23:20:18.812125 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:20:18.812091    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:20:21.189580 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.189550    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"]
Nov 01 23:20:21.190036 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.189943    4837 cpu_manager.go:414] "containerMap: removing container" podUID="b4d71455-c93a-4c15-bd57-d46e600703fc" containerName="revoke-keys"
Nov 01 23:20:21.190036 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.189953    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d71455-c93a-4c15-bd57-d46e600703fc" containerName="revoke-keys"
Nov 01 23:20:21.190036 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.190031    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d71455-c93a-4c15-bd57-d46e600703fc" containerName="revoke-keys"
Nov 01 23:20:21.193354 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.193332    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.243781 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.243747    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:20:21.290237 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.290203    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:20:21.290237 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.290223    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:20:21.290460 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.290268    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:20:21.292601 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.292578    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"]
Nov 01 23:20:21.371487 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.371459    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.371631 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.371507    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrztl\" (UniqueName: \"kubernetes.io/projected/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-kube-api-access-xrztl\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.371676 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.371660    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.473184 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.473121    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.473316 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.473184    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.473316 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.473226    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrztl\" (UniqueName: \"kubernetes.io/projected/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-kube-api-access-xrztl\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.473652 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.473625    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.475436 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.475416    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.526309 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.526285    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrztl\" (UniqueName: \"kubernetes.io/projected/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-kube-api-access-xrztl\") pod \"maas-api-revoke-sub-3d4156f1-4b57z\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:21.806285 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:21.806217    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:22.168831 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:22.168797    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"]
Nov 01 23:20:22.169201 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:20:22.169172    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1cfeb8_d5d9_44c9_9855_c399358cf9f0.slice/crio-87c5c557b560fbc6a87bd2f7401201cec0b420a5e2d36ae6569642cd912ca230 WatchSource:0}: Error finding container 87c5c557b560fbc6a87bd2f7401201cec0b420a5e2d36ae6569642cd912ca230: Status 404 returned error can't find the container with id 87c5c557b560fbc6a87bd2f7401201cec0b420a5e2d36ae6569642cd912ca230
Nov 01 23:20:22.481934 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:22.481900    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:20:23.116065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:23.098929    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z" event={"ID":"df1cfeb8-d5d9-44c9-9855-c399358cf9f0","Type":"ContainerStarted","Data":"e3f257ecc7bfc1323ca8084f08edfc87c5a2a2e261c2ebcdf0596f484daff82f"}
Nov 01 23:20:23.116065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:23.099034    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z" event={"ID":"df1cfeb8-d5d9-44c9-9855-c399358cf9f0","Type":"ContainerStarted","Data":"87c5c557b560fbc6a87bd2f7401201cec0b420a5e2d36ae6569642cd912ca230"}
Nov 01 23:20:24.115279 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:24.115228    4837 generic.go:356] "Generic (PLEG): container finished" podID="df1cfeb8-d5d9-44c9-9855-c399358cf9f0" containerID="e3f257ecc7bfc1323ca8084f08edfc87c5a2a2e261c2ebcdf0596f484daff82f" exitCode=0
Nov 01 23:20:24.115977 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:24.115923    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z" event={"ID":"df1cfeb8-d5d9-44c9-9855-c399358cf9f0","Type":"ContainerDied","Data":"e3f257ecc7bfc1323ca8084f08edfc87c5a2a2e261c2ebcdf0596f484daff82f"}
Nov 01 23:20:25.261105 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.261082    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:25.343082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.343030    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-xks-ca\") pod \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") "
Nov 01 23:20:25.343227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.343099    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-openshift-ca\") pod \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") "
Nov 01 23:20:25.343227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.343214    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-kube-api-access-xrztl\" (UniqueName: \"kubernetes.io/projected/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-kube-api-access-xrztl\") pod \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\" (UID: \"df1cfeb8-d5d9-44c9-9855-c399358cf9f0\") "
Nov 01 23:20:25.343466 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.343439    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-openshift-ca" pod "df1cfeb8-d5d9-44c9-9855-c399358cf9f0" (UID: "df1cfeb8-d5d9-44c9-9855-c399358cf9f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:20:25.345102 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.345072    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-xks-ca" pod "df1cfeb8-d5d9-44c9-9855-c399358cf9f0" (UID: "df1cfeb8-d5d9-44c9-9855-c399358cf9f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:20:25.345221 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.345198    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-kube-api-access-xrztl" pod "df1cfeb8-d5d9-44c9-9855-c399358cf9f0" (UID: "df1cfeb8-d5d9-44c9-9855-c399358cf9f0"). InnerVolumeSpecName "kube-api-access-xrztl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:20:25.444775 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.444753    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:25.444775 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.444777    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:25.444945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:25.444787    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrztl\" (UniqueName: \"kubernetes.io/projected/df1cfeb8-d5d9-44c9-9855-c399358cf9f0-kube-api-access-xrztl\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:26.127271 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:26.127241    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"
Nov 01 23:20:26.127462 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:26.127282    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z" event={"ID":"df1cfeb8-d5d9-44c9-9855-c399358cf9f0","Type":"ContainerDied","Data":"87c5c557b560fbc6a87bd2f7401201cec0b420a5e2d36ae6569642cd912ca230"}
Nov 01 23:20:26.127462 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:26.127339    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87c5c557b560fbc6a87bd2f7401201cec0b420a5e2d36ae6569642cd912ca230"
Nov 01 23:20:38.043961 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:38.043928    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"]
Nov 01 23:20:38.090656 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:38.090631    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-wjbl2"]
Nov 01 23:20:38.523124 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:38.523091    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3caa607-fece-448d-aecd-727b553353aa" path="/var/lib/kubelet/pods/a3caa607-fece-448d-aecd-727b553353aa/volumes"
Nov 01 23:20:39.117723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.117692    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"]
Nov 01 23:20:39.164247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.164218    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4b57z"]
Nov 01 23:20:39.237847 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.237822    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"]
Nov 01 23:20:39.238232 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.238220    4837 cpu_manager.go:414] "containerMap: removing container" podUID="df1cfeb8-d5d9-44c9-9855-c399358cf9f0" containerName="revoke-keys"
Nov 01 23:20:39.238286 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.238233    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1cfeb8-d5d9-44c9-9855-c399358cf9f0" containerName="revoke-keys"
Nov 01 23:20:39.238324 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.238315    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1cfeb8-d5d9-44c9-9855-c399358cf9f0" containerName="revoke-keys"
Nov 01 23:20:39.241461 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.241443    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.265188 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.265166    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsvzk\" (UniqueName: \"kubernetes.io/projected/4c836d8c-4df3-449a-9bb0-ac5d909d0169-kube-api-access-gsvzk\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.265298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.265209    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.265298 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.265273    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.290597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.290566    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:20:39.290597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.290592    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:20:39.290745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.290646    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:20:39.290745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.290656    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:20:39.296267 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.296239    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"]
Nov 01 23:20:39.366006 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.365981    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.366143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.366020    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.366143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.366130    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsvzk\" (UniqueName: \"kubernetes.io/projected/4c836d8c-4df3-449a-9bb0-ac5d909d0169-kube-api-access-gsvzk\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.366709 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.366687    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.368318 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.368280    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.419227 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.419202    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsvzk\" (UniqueName: \"kubernetes.io/projected/4c836d8c-4df3-449a-9bb0-ac5d909d0169-kube-api-access-gsvzk\") pod \"maas-api-revoke-sub-3d4156f1-h75vl\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.554308 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.554285    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:39.916442 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:39.916413    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"]
Nov 01 23:20:39.917455 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:20:39.917429    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c836d8c_4df3_449a_9bb0_ac5d909d0169.slice/crio-467c0c15c5a98bc6eb6c6071c84dddb48dc5c2363ca0d537787d2a19c9e5487d WatchSource:0}: Error finding container 467c0c15c5a98bc6eb6c6071c84dddb48dc5c2363ca0d537787d2a19c9e5487d: Status 404 returned error can't find the container with id 467c0c15c5a98bc6eb6c6071c84dddb48dc5c2363ca0d537787d2a19c9e5487d
Nov 01 23:20:40.203105 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:40.203079    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl" event={"ID":"4c836d8c-4df3-449a-9bb0-ac5d909d0169","Type":"ContainerStarted","Data":"467c0c15c5a98bc6eb6c6071c84dddb48dc5c2363ca0d537787d2a19c9e5487d"}
Nov 01 23:20:40.312258 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:40.312233    4837 scope.go:122] "RemoveContainer" containerID="e6ca0d10c3b80171e4938c586f61739cd95ccb3414c3942a9ec7ee64fc7b5fb0"
Nov 01 23:20:40.322887 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:40.322867    4837 scope.go:122] "RemoveContainer" containerID="9beb4e4a27a254fc925f941e7ec258c061ae04b5d683b3f8f8a61a4f96861876"
Nov 01 23:20:40.330756 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:40.330730    4837 scope.go:122] "RemoveContainer" containerID="821e543429e985f9da89c13924ff79e37dc3ed1b07ea8c048d4b7f90e41c8f65"
Nov 01 23:20:40.338239 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:40.338219    4837 scope.go:122] "RemoveContainer" containerID="f049cda2e85b94723daaf7f8861ddd4abde2464ccbbf2ad12b6067738734f556"
Nov 01 23:20:40.524142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:40.524070    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1cfeb8-d5d9-44c9-9855-c399358cf9f0" path="/var/lib/kubelet/pods/df1cfeb8-d5d9-44c9-9855-c399358cf9f0/volumes"
Nov 01 23:20:41.209365 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:41.209327    4837 generic.go:356] "Generic (PLEG): container finished" podID="4c836d8c-4df3-449a-9bb0-ac5d909d0169" containerID="7e74bfcf2d5da6a29dea5963bb60f7bc1e00fcf0746f0c08f763e41c4a81f0fc" exitCode=0
Nov 01 23:20:41.209784 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:41.209417    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl" event={"ID":"4c836d8c-4df3-449a-9bb0-ac5d909d0169","Type":"ContainerDied","Data":"7e74bfcf2d5da6a29dea5963bb60f7bc1e00fcf0746f0c08f763e41c4a81f0fc"}
Nov 01 23:20:42.349531 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.349508    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:42.395833 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.395806    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-openshift-ca\") pod \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") "
Nov 01 23:20:42.395988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.395858    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c836d8c-4df3-449a-9bb0-ac5d909d0169-kube-api-access-gsvzk\" (UniqueName: \"kubernetes.io/projected/4c836d8c-4df3-449a-9bb0-ac5d909d0169-kube-api-access-gsvzk\") pod \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") "
Nov 01 23:20:42.395988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.395910    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-xks-ca\") pod \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\" (UID: \"4c836d8c-4df3-449a-9bb0-ac5d909d0169\") "
Nov 01 23:20:42.396199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.396170    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-openshift-ca" pod "4c836d8c-4df3-449a-9bb0-ac5d909d0169" (UID: "4c836d8c-4df3-449a-9bb0-ac5d909d0169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:20:42.397970 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.397944    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c836d8c-4df3-449a-9bb0-ac5d909d0169-kube-api-access-gsvzk" pod "4c836d8c-4df3-449a-9bb0-ac5d909d0169" (UID: "4c836d8c-4df3-449a-9bb0-ac5d909d0169"). InnerVolumeSpecName "kube-api-access-gsvzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:20:42.398118 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.397998    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-xks-ca" pod "4c836d8c-4df3-449a-9bb0-ac5d909d0169" (UID: "4c836d8c-4df3-449a-9bb0-ac5d909d0169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:20:42.497341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.497269    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:42.497341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.497295    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsvzk\" (UniqueName: \"kubernetes.io/projected/4c836d8c-4df3-449a-9bb0-ac5d909d0169-kube-api-access-gsvzk\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:42.497341 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:42.497307    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c836d8c-4df3-449a-9bb0-ac5d909d0169-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:20:43.221214 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:43.221185    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"
Nov 01 23:20:43.221405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:43.221227    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl" event={"ID":"4c836d8c-4df3-449a-9bb0-ac5d909d0169","Type":"ContainerDied","Data":"467c0c15c5a98bc6eb6c6071c84dddb48dc5c2363ca0d537787d2a19c9e5487d"}
Nov 01 23:20:43.221405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:20:43.221273    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="467c0c15c5a98bc6eb6c6071c84dddb48dc5c2363ca0d537787d2a19c9e5487d"
Nov 01 23:20:48.811621 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:20:48.811585    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:21:05.081166 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:05.081130    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:21:13.842526 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.842486    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"]
Nov 01 23:21:13.843264 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.843238    4837 cpu_manager.go:414] "containerMap: removing container" podUID="4c836d8c-4df3-449a-9bb0-ac5d909d0169" containerName="revoke-keys"
Nov 01 23:21:13.843264 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.843262    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c836d8c-4df3-449a-9bb0-ac5d909d0169" containerName="revoke-keys"
Nov 01 23:21:13.843470 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.843401    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c836d8c-4df3-449a-9bb0-ac5d909d0169" containerName="revoke-keys"
Nov 01 23:21:13.846393 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.846377    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:13.885715 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.885686    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:13.885855 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.885734    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:13.885855 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.885777    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdcz\" (UniqueName: \"kubernetes.io/projected/8be3de8a-aedc-45fe-b4ed-edc9243b525e-kube-api-access-dhdcz\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:13.900293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.900264    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"]
Nov 01 23:21:13.943395 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.943365    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:21:13.943554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.943461    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:21:13.943554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.943461    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:21:13.943554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.943488    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:21:13.987212 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.987183    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdcz\" (UniqueName: \"kubernetes.io/projected/8be3de8a-aedc-45fe-b4ed-edc9243b525e-kube-api-access-dhdcz\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:13.987357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.987285    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:13.987357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.987335    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:13.987945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.987925    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:13.989549 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:13.989529    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:14.040532 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:14.040513    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdcz\" (UniqueName: \"kubernetes.io/projected/8be3de8a-aedc-45fe-b4ed-edc9243b525e-kube-api-access-dhdcz\") pod \"maas-api-revoke-sub-e2fc9c5b-t8jpz\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:14.158973 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:14.158916    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:14.524809 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:14.524785    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"]
Nov 01 23:21:14.526491 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:21:14.526459    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be3de8a_aedc_45fe_b4ed_edc9243b525e.slice/crio-96e48acaa6f0df34d50b22de71520ef62da7198ddfae40f75caa069079b1695c WatchSource:0}: Error finding container 96e48acaa6f0df34d50b22de71520ef62da7198ddfae40f75caa069079b1695c: Status 404 returned error can't find the container with id 96e48acaa6f0df34d50b22de71520ef62da7198ddfae40f75caa069079b1695c
Nov 01 23:21:15.416273 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:15.416237    4837 generic.go:356] "Generic (PLEG): container finished" podID="8be3de8a-aedc-45fe-b4ed-edc9243b525e" containerID="7603d024e10d9fb3827b6cc1054583395dd4ff706b313b4cd4820d97cdf7d85f" exitCode=0
Nov 01 23:21:15.416630 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:15.416323    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz" event={"ID":"8be3de8a-aedc-45fe-b4ed-edc9243b525e","Type":"ContainerDied","Data":"7603d024e10d9fb3827b6cc1054583395dd4ff706b313b4cd4820d97cdf7d85f"}
Nov 01 23:21:15.416630 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:15.416373    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz" event={"ID":"8be3de8a-aedc-45fe-b4ed-edc9243b525e","Type":"ContainerStarted","Data":"96e48acaa6f0df34d50b22de71520ef62da7198ddfae40f75caa069079b1695c"}
Nov 01 23:21:16.570292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.570270    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:16.612554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.612530    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-xks-ca\") pod \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") "
Nov 01 23:21:16.612673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.612583    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-openshift-ca\") pod \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") "
Nov 01 23:21:16.612673 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.612652    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be3de8a-aedc-45fe-b4ed-edc9243b525e-kube-api-access-dhdcz\" (UniqueName: \"kubernetes.io/projected/8be3de8a-aedc-45fe-b4ed-edc9243b525e-kube-api-access-dhdcz\") pod \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\" (UID: \"8be3de8a-aedc-45fe-b4ed-edc9243b525e\") "
Nov 01 23:21:16.612959 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.612935    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-openshift-ca" pod "8be3de8a-aedc-45fe-b4ed-edc9243b525e" (UID: "8be3de8a-aedc-45fe-b4ed-edc9243b525e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:21:16.614592 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.614572    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-xks-ca" pod "8be3de8a-aedc-45fe-b4ed-edc9243b525e" (UID: "8be3de8a-aedc-45fe-b4ed-edc9243b525e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:21:16.614681 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.614668    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be3de8a-aedc-45fe-b4ed-edc9243b525e-kube-api-access-dhdcz" pod "8be3de8a-aedc-45fe-b4ed-edc9243b525e" (UID: "8be3de8a-aedc-45fe-b4ed-edc9243b525e"). InnerVolumeSpecName "kube-api-access-dhdcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:21:16.713471 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.713410    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:16.713471 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.713435    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be3de8a-aedc-45fe-b4ed-edc9243b525e-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:16.713471 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:16.713445    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhdcz\" (UniqueName: \"kubernetes.io/projected/8be3de8a-aedc-45fe-b4ed-edc9243b525e-kube-api-access-dhdcz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:17.435240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:17.435203    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"
Nov 01 23:21:17.435240 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:17.435218    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz" event={"ID":"8be3de8a-aedc-45fe-b4ed-edc9243b525e","Type":"ContainerDied","Data":"96e48acaa6f0df34d50b22de71520ef62da7198ddfae40f75caa069079b1695c"}
Nov 01 23:21:17.435454 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:17.435291    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96e48acaa6f0df34d50b22de71520ef62da7198ddfae40f75caa069079b1695c"
Nov 01 23:21:18.811907 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:21:18.811864    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:21:24.621084 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.621036    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"]
Nov 01 23:21:24.621494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.621475    4837 cpu_manager.go:414] "containerMap: removing container" podUID="8be3de8a-aedc-45fe-b4ed-edc9243b525e" containerName="revoke-keys"
Nov 01 23:21:24.621494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.621485    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be3de8a-aedc-45fe-b4ed-edc9243b525e" containerName="revoke-keys"
Nov 01 23:21:24.621584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.621561    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be3de8a-aedc-45fe-b4ed-edc9243b525e" containerName="revoke-keys"
Nov 01 23:21:24.624992 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.624973    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.679865 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.679827    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"]
Nov 01 23:21:24.689077 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.689030    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgfd2\" (UniqueName: \"kubernetes.io/projected/d88194a3-dcd1-4655-a5af-dba65f7f7dff-kube-api-access-tgfd2\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.689223 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.689131    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.689223 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.689200    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.721969 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.721945    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:21:24.722110 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.721972    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:21:24.722110 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.721952    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:21:24.722110 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.722007    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:21:24.790462 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.790435    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfd2\" (UniqueName: \"kubernetes.io/projected/d88194a3-dcd1-4655-a5af-dba65f7f7dff-kube-api-access-tgfd2\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.790577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.790466    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.790577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.790498    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.791113 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.791096    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.792863 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.792846    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.843635 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.843609    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfd2\" (UniqueName: \"kubernetes.io/projected/d88194a3-dcd1-4655-a5af-dba65f7f7dff-kube-api-access-tgfd2\") pod \"maas-api-revoke-sub-5a29efb3-fkstw\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:24.941509 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:24.941491    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:25.305162 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:25.305134    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"]
Nov 01 23:21:25.306235 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:21:25.306204    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88194a3_dcd1_4655_a5af_dba65f7f7dff.slice/crio-af793e0d44fb7c777ae9049df2ab17e6f9f3c1ddc75e8be26c15c4514ec4bfd1 WatchSource:0}: Error finding container af793e0d44fb7c777ae9049df2ab17e6f9f3c1ddc75e8be26c15c4514ec4bfd1: Status 404 returned error can't find the container with id af793e0d44fb7c777ae9049df2ab17e6f9f3c1ddc75e8be26c15c4514ec4bfd1
Nov 01 23:21:25.477475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:25.477450    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw" event={"ID":"d88194a3-dcd1-4655-a5af-dba65f7f7dff","Type":"ContainerStarted","Data":"af793e0d44fb7c777ae9049df2ab17e6f9f3c1ddc75e8be26c15c4514ec4bfd1"}
Nov 01 23:21:26.483911 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:26.483881    4837 generic.go:356] "Generic (PLEG): container finished" podID="d88194a3-dcd1-4655-a5af-dba65f7f7dff" containerID="a7588d9ba8a2b8cb5d77b9bf2c7f0a10c186bfe13d6a478d006fa05304f71100" exitCode=0
Nov 01 23:21:26.484616 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:26.483973    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw" event={"ID":"d88194a3-dcd1-4655-a5af-dba65f7f7dff","Type":"ContainerDied","Data":"a7588d9ba8a2b8cb5d77b9bf2c7f0a10c186bfe13d6a478d006fa05304f71100"}
Nov 01 23:21:26.581604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:26.581572    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:21:27.681963 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:27.681932    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:27.853119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:27.845228    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-openshift-ca\") pod \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") "
Nov 01 23:21:27.853119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:27.845284    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88194a3-dcd1-4655-a5af-dba65f7f7dff-kube-api-access-tgfd2\" (UniqueName: \"kubernetes.io/projected/d88194a3-dcd1-4655-a5af-dba65f7f7dff-kube-api-access-tgfd2\") pod \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") "
Nov 01 23:21:27.853119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:27.845344    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-xks-ca\") pod \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\" (UID: \"d88194a3-dcd1-4655-a5af-dba65f7f7dff\") "
Nov 01 23:21:27.932306 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:27.888629    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-openshift-ca" pod "d88194a3-dcd1-4655-a5af-dba65f7f7dff" (UID: "d88194a3-dcd1-4655-a5af-dba65f7f7dff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:21:27.969066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:27.961748    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:27.969066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:27.967199    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88194a3-dcd1-4655-a5af-dba65f7f7dff-kube-api-access-tgfd2" pod "d88194a3-dcd1-4655-a5af-dba65f7f7dff" (UID: "d88194a3-dcd1-4655-a5af-dba65f7f7dff"). InnerVolumeSpecName "kube-api-access-tgfd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:21:28.091091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:28.038175    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-xks-ca" pod "d88194a3-dcd1-4655-a5af-dba65f7f7dff" (UID: "d88194a3-dcd1-4655-a5af-dba65f7f7dff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:21:28.091091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:28.066288    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgfd2\" (UniqueName: \"kubernetes.io/projected/d88194a3-dcd1-4655-a5af-dba65f7f7dff-kube-api-access-tgfd2\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:28.091091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:28.066328    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d88194a3-dcd1-4655-a5af-dba65f7f7dff-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:28.628066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:28.627212    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"
Nov 01 23:21:28.647066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:28.628628    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw" event={"ID":"d88194a3-dcd1-4655-a5af-dba65f7f7dff","Type":"ContainerDied","Data":"af793e0d44fb7c777ae9049df2ab17e6f9f3c1ddc75e8be26c15c4514ec4bfd1"}
Nov 01 23:21:28.647066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:28.628688    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af793e0d44fb7c777ae9049df2ab17e6f9f3c1ddc75e8be26c15c4514ec4bfd1"
Nov 01 23:21:37.268309 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:37.268275    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:21:40.375758 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:40.375728    4837 scope.go:122] "RemoveContainer" containerID="27550a61b66eaeffe7ae3334c258bf7cac86bf9fdc3112eb6230c71b5ef14929"
Nov 01 23:21:41.199510 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.199477    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"]
Nov 01 23:21:41.242994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.242965    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h75vl"]
Nov 01 23:21:41.320069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.320024    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"]
Nov 01 23:21:41.320484 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.320471    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d88194a3-dcd1-4655-a5af-dba65f7f7dff" containerName="revoke-keys"
Nov 01 23:21:41.320530 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.320486    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88194a3-dcd1-4655-a5af-dba65f7f7dff" containerName="revoke-keys"
Nov 01 23:21:41.320566 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.320547    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88194a3-dcd1-4655-a5af-dba65f7f7dff" containerName="revoke-keys"
Nov 01 23:21:41.325256 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.325237    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.375228 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.375185    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:21:41.375228 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.375202    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:21:41.375437 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.375272    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:21:41.375437 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.375303    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:21:41.378438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.378413    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"]
Nov 01 23:21:41.397886 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.397859    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.398024 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.397991    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.398102 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.398027    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48qh5\" (UniqueName: \"kubernetes.io/projected/ff37b204-f780-49ba-9605-d12d65c58293-kube-api-access-48qh5\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.499556 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.499494    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.499679 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.499556    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.499679 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.499582    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48qh5\" (UniqueName: \"kubernetes.io/projected/ff37b204-f780-49ba-9605-d12d65c58293-kube-api-access-48qh5\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.500281 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.500258    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.501983 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.501959    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.553347 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.553321    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48qh5\" (UniqueName: \"kubernetes.io/projected/ff37b204-f780-49ba-9605-d12d65c58293-kube-api-access-48qh5\") pod \"maas-api-revoke-sub-3d4156f1-6zgnj\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:41.638427 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:41.638404    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:42.004164 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:42.004141    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"]
Nov 01 23:21:42.006182 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:21:42.006153    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff37b204_f780_49ba_9605_d12d65c58293.slice/crio-1191d2117d48a6b8b7f78e01d84dea829a70ae7ddad28a39e23982904ba084b8 WatchSource:0}: Error finding container 1191d2117d48a6b8b7f78e01d84dea829a70ae7ddad28a39e23982904ba084b8: Status 404 returned error can't find the container with id 1191d2117d48a6b8b7f78e01d84dea829a70ae7ddad28a39e23982904ba084b8
Nov 01 23:21:42.524432 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:42.524405    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c836d8c-4df3-449a-9bb0-ac5d909d0169" path="/var/lib/kubelet/pods/4c836d8c-4df3-449a-9bb0-ac5d909d0169/volumes"
Nov 01 23:21:42.683008 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:42.682978    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj" event={"ID":"ff37b204-f780-49ba-9605-d12d65c58293","Type":"ContainerStarted","Data":"09c9f189a5e1409695f74d5a7c588f8874715a8418b926cbd3b895f34e2eb99d"}
Nov 01 23:21:42.683182 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:42.683060    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj" event={"ID":"ff37b204-f780-49ba-9605-d12d65c58293","Type":"ContainerStarted","Data":"1191d2117d48a6b8b7f78e01d84dea829a70ae7ddad28a39e23982904ba084b8"}
Nov 01 23:21:42.791177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:42.791130    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj" podStartSLOduration=1.791116439 podStartE2EDuration="1.791116439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:21:42.788950463 +0000 UTC m=+3484.378149528" watchObservedRunningTime="2026-11-01 23:21:42.791116439 +0000 UTC m=+3484.380315541"
Nov 01 23:21:43.689729 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:43.689693    4837 generic.go:356] "Generic (PLEG): container finished" podID="ff37b204-f780-49ba-9605-d12d65c58293" containerID="09c9f189a5e1409695f74d5a7c588f8874715a8418b926cbd3b895f34e2eb99d" exitCode=0
Nov 01 23:21:43.690233 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:43.689778    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj" event={"ID":"ff37b204-f780-49ba-9605-d12d65c58293","Type":"ContainerDied","Data":"09c9f189a5e1409695f74d5a7c588f8874715a8418b926cbd3b895f34e2eb99d"}
Nov 01 23:21:44.835836 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:44.835809    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:44.931584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:44.931555    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-xks-ca\") pod \"ff37b204-f780-49ba-9605-d12d65c58293\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") "
Nov 01 23:21:44.931770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:44.931635    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-openshift-ca\") pod \"ff37b204-f780-49ba-9605-d12d65c58293\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") "
Nov 01 23:21:44.931770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:44.931742    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff37b204-f780-49ba-9605-d12d65c58293-kube-api-access-48qh5\" (UniqueName: \"kubernetes.io/projected/ff37b204-f780-49ba-9605-d12d65c58293-kube-api-access-48qh5\") pod \"ff37b204-f780-49ba-9605-d12d65c58293\" (UID: \"ff37b204-f780-49ba-9605-d12d65c58293\") "
Nov 01 23:21:44.931999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:44.931966    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-openshift-ca" pod "ff37b204-f780-49ba-9605-d12d65c58293" (UID: "ff37b204-f780-49ba-9605-d12d65c58293"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:21:44.932186 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:44.932168    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:44.933747 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:44.933723    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-xks-ca" pod "ff37b204-f780-49ba-9605-d12d65c58293" (UID: "ff37b204-f780-49ba-9605-d12d65c58293"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:21:44.933916 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:44.933901    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff37b204-f780-49ba-9605-d12d65c58293-kube-api-access-48qh5" pod "ff37b204-f780-49ba-9605-d12d65c58293" (UID: "ff37b204-f780-49ba-9605-d12d65c58293"). InnerVolumeSpecName "kube-api-access-48qh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:21:45.033222 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:45.033156    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff37b204-f780-49ba-9605-d12d65c58293-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:45.033222 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:45.033183    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48qh5\" (UniqueName: \"kubernetes.io/projected/ff37b204-f780-49ba-9605-d12d65c58293-kube-api-access-48qh5\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:21:45.701610 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:45.701578    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"
Nov 01 23:21:45.701610 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:45.701601    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj" event={"ID":"ff37b204-f780-49ba-9605-d12d65c58293","Type":"ContainerDied","Data":"1191d2117d48a6b8b7f78e01d84dea829a70ae7ddad28a39e23982904ba084b8"}
Nov 01 23:21:45.701832 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:45.701650    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1191d2117d48a6b8b7f78e01d84dea829a70ae7ddad28a39e23982904ba084b8"
Nov 01 23:21:46.049779 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:46.049702    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"]
Nov 01 23:21:46.052441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:46.052420    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9vt4b"]
Nov 01 23:21:46.523007 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:21:46.522975    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9822fc96-e7f4-4822-b788-81049586e067" path="/var/lib/kubelet/pods/9822fc96-e7f4-4822-b788-81049586e067/volumes"
Nov 01 23:21:48.811194 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:21:48.811157    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:22:05.729678 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.729644    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"]
Nov 01 23:22:05.730231 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.730069    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ff37b204-f780-49ba-9605-d12d65c58293" containerName="revoke-keys"
Nov 01 23:22:05.730231 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.730081    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff37b204-f780-49ba-9605-d12d65c58293" containerName="revoke-keys"
Nov 01 23:22:05.730231 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.730169    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff37b204-f780-49ba-9605-d12d65c58293" containerName="revoke-keys"
Nov 01 23:22:05.734754 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.734735    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:05.786871 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.786844    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3fc07c\""
Nov 01 23:22:05.787085 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.787031    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c1bca7b9-dockercfg-4dd7p\""
Nov 01 23:22:05.832512 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.832481    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"]
Nov 01 23:22:05.839449 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.839426    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slndp\" (UniqueName: \"kubernetes.io/projected/9f6cb910-1c84-4b34-828a-04a2716511d8-kube-api-access-slndp\") pod \"payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w\" (UID: \"9f6cb910-1c84-4b34-828a-04a2716511d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:05.839581 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.839565    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6cb910-1c84-4b34-828a-04a2716511d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w\" (UID: \"9f6cb910-1c84-4b34-828a-04a2716511d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:05.940995 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.940964    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6cb910-1c84-4b34-828a-04a2716511d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w\" (UID: \"9f6cb910-1c84-4b34-828a-04a2716511d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:05.941213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.941015    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slndp\" (UniqueName: \"kubernetes.io/projected/9f6cb910-1c84-4b34-828a-04a2716511d8-kube-api-access-slndp\") pod \"payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w\" (UID: \"9f6cb910-1c84-4b34-828a-04a2716511d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:05.941755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.941734    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6cb910-1c84-4b34-828a-04a2716511d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w\" (UID: \"9f6cb910-1c84-4b34-828a-04a2716511d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:05.996331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:05.996273    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slndp\" (UniqueName: \"kubernetes.io/projected/9f6cb910-1c84-4b34-828a-04a2716511d8-kube-api-access-slndp\") pod \"payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w\" (UID: \"9f6cb910-1c84-4b34-828a-04a2716511d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:06.047421 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:06.047396    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:06.416702 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:06.416680    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"]
Nov 01 23:22:06.418823 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:22:06.418796    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6cb910_1c84_4b34_828a_04a2716511d8.slice/crio-24894389973d373cd6d132e1d9414546a4aad2c6f6648a33176f5fc14ce92e82 WatchSource:0}: Error finding container 24894389973d373cd6d132e1d9414546a4aad2c6f6648a33176f5fc14ce92e82: Status 404 returned error can't find the container with id 24894389973d373cd6d132e1d9414546a4aad2c6f6648a33176f5fc14ce92e82
Nov 01 23:22:06.815752 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:06.815714    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w" event={"ID":"9f6cb910-1c84-4b34-828a-04a2716511d8","Type":"ContainerStarted","Data":"d400596fc696fbda163745eecd07b6bd5835d1f6a5d5924b5a4138fac08c0eb6"}
Nov 01 23:22:06.816287 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:06.815770    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:06.816287 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:06.815788    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w" event={"ID":"9f6cb910-1c84-4b34-828a-04a2716511d8","Type":"ContainerStarted","Data":"24894389973d373cd6d132e1d9414546a4aad2c6f6648a33176f5fc14ce92e82"}
Nov 01 23:22:06.930653 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:06.930605    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w" podStartSLOduration=1.930592233 podStartE2EDuration="1.930592233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:22:06.929114912 +0000 UTC m=+3508.518313976" watchObservedRunningTime="2026-11-01 23:22:06.930592233 +0000 UTC m=+3508.519791336"
Nov 01 23:22:07.786590 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.786556    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"]
Nov 01 23:22:07.790030 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.790012    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:07.885515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.885489    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:22:07.885515 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.885489    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:22:07.885926 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.885493    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3fc07c-serving-cert\""
Nov 01 23:22:07.885926 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.885539    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xp822\""
Nov 01 23:22:07.885926 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.885495    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 23:22:07.887991 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.887974    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"]
Nov 01 23:22:07.961723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.961697    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:07.961879 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.961805    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cd247f-abb5-4903-8c21-21d8caf8d110-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:07.961970 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.961953    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4ksr\" (UniqueName: \"kubernetes.io/projected/b6cd247f-abb5-4903-8c21-21d8caf8d110-kube-api-access-j4ksr\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:07.962015 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:07.961999    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.062994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.062921    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cd247f-abb5-4903-8c21-21d8caf8d110-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.063177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.063011    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4ksr\" (UniqueName: \"kubernetes.io/projected/b6cd247f-abb5-4903-8c21-21d8caf8d110-kube-api-access-j4ksr\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.063177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.063076    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.063177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.063114    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.063291 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:22:08.063200    4837 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-serving-cert: secret "maas-api-e2e-worker-single-main-3fc07c-serving-cert" not found
Nov 01 23:22:08.063291 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:22:08.063277    4837 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls podName:b6cd247f-abb5-4903-8c21-21d8caf8d110 nodeName:}" failed. No retries permitted until 2026-11-01 23:22:08.563256084 +0000 UTC m=+3510.152455139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls") pod "maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm" (UID: "b6cd247f-abb5-4903-8c21-21d8caf8d110") : secret "maas-api-e2e-worker-single-main-3fc07c-serving-cert" not found
Nov 01 23:22:08.063684 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.063665    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cd247f-abb5-4903-8c21-21d8caf8d110-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.065573 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.065554    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.119112 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.119087    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4ksr\" (UniqueName: \"kubernetes.io/projected/b6cd247f-abb5-4903-8c21-21d8caf8d110-kube-api-access-j4ksr\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.569689 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.569658    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.571957 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.571931    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:08.702322 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:08.702290    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:09.065531 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:09.065504    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"]
Nov 01 23:22:09.066455 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:22:09.066424    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cd247f_abb5_4903_8c21_21d8caf8d110.slice/crio-34cf95d40cfae11228f63715ec19ea4e95f4cd48302bceeac4089f73fdd84cb6 WatchSource:0}: Error finding container 34cf95d40cfae11228f63715ec19ea4e95f4cd48302bceeac4089f73fdd84cb6: Status 404 returned error can't find the container with id 34cf95d40cfae11228f63715ec19ea4e95f4cd48302bceeac4089f73fdd84cb6
Nov 01 23:22:09.068091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:09.068073    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 23:22:09.832792 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:09.832759    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm" event={"ID":"b6cd247f-abb5-4903-8c21-21d8caf8d110","Type":"ContainerStarted","Data":"34cf95d40cfae11228f63715ec19ea4e95f4cd48302bceeac4089f73fdd84cb6"}
Nov 01 23:22:10.840079 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:10.840022    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm" event={"ID":"b6cd247f-abb5-4903-8c21-21d8caf8d110","Type":"ContainerStarted","Data":"d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b"}
Nov 01 23:22:10.840542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:10.840130    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:10.948142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:10.948093    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm" podStartSLOduration=2.881282475 podStartE2EDuration="3.9480797s" totalImagesPullingTime="1.066797225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:22:10.947156297 +0000 UTC m=+3512.536355359" watchObservedRunningTime="2026-11-01 23:22:10.9480797 +0000 UTC m=+3512.537278761"
Nov 01 23:22:15.672829 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:15.672789    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:22:16.851026 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:16.850993    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:22:17.824276 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:17.824248    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:22:18.045376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:18.045345    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"]
Nov 01 23:22:18.047816 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:18.047792    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9ktd8"]
Nov 01 23:22:18.523968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:18.523940    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85538e02-5dc7-4d77-b3fd-4b2e937db384" path="/var/lib/kubelet/pods/85538e02-5dc7-4d77-b3fd-4b2e937db384/volumes"
Nov 01 23:22:18.811943 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:22:18.811858    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:22:31.046126 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:31.045976    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"]
Nov 01 23:22:31.091982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:31.091949    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-txtcf"]
Nov 01 23:22:32.523023 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:32.522988    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5591f338-1d2b-4ef4-aa62-3998830f7cae" path="/var/lib/kubelet/pods/5591f338-1d2b-4ef4-aa62-3998830f7cae/volumes"
Nov 01 23:22:34.763879 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:22:34.763849    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:22:48.811794 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:22:48.811707    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:23:05.046542 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:05.046505    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"]
Nov 01 23:23:05.093013 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:05.092977    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wzlzc"]
Nov 01 23:23:05.156099 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:05.156061    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:23:06.523598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:06.523560    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67daa55-b5ea-4ac5-aded-b9f125ffd400" path="/var/lib/kubelet/pods/d67daa55-b5ea-4ac5-aded-b9f125ffd400/volumes"
Nov 01 23:23:18.044629 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:18.044591    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"]
Nov 01 23:23:18.091188 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:18.091153    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zf2bz"]
Nov 01 23:23:18.523962 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:18.523933    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a0daad-4b35-4ea6-929f-73888dcf21f5" path="/var/lib/kubelet/pods/14a0daad-4b35-4ea6-929f-73888dcf21f5/volumes"
Nov 01 23:23:18.812192 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:23:18.812109    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:23:34.159198 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:34.159157    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:23:39.040408 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:39.040378    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:23:39.042691 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:39.042667    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:23:40.406883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:40.406852    4837 scope.go:122] "RemoveContainer" containerID="de487a46e0f3c2d0d185e70d60e66cd5248e585e685133506812cc205040e87e"
Nov 01 23:23:40.415536 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:40.415516    4837 scope.go:122] "RemoveContainer" containerID="57bac4b5a90c9c9ca7dc9e5bb8b0ec8bca09d8196d605fff11eb5dc932747ad4"
Nov 01 23:23:40.424009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:40.423991    4837 scope.go:122] "RemoveContainer" containerID="a9ee881b1e2ce776cb9dce363059f938e56c8427bc566c1b2a67af917c8ab5d9"
Nov 01 23:23:43.280856 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:43.280817    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"]
Nov 01 23:23:44.650104 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:44.650063    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"]
Nov 01 23:23:44.650620 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:44.650381    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm" podUID="b6cd247f-abb5-4903-8c21-21d8caf8d110" containerName="maas-api" containerID="cri-o://d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b" gracePeriod=30
Nov 01 23:23:45.143238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.143208    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:23:45.257307 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.257225    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-metrics-certs\") pod \"b6cd247f-abb5-4903-8c21-21d8caf8d110\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") "
Nov 01 23:23:45.257483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.257325    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls\") pod \"b6cd247f-abb5-4903-8c21-21d8caf8d110\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") "
Nov 01 23:23:45.257483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.257380    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cd247f-abb5-4903-8c21-21d8caf8d110-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cd247f-abb5-4903-8c21-21d8caf8d110-openshift-service-ca\") pod \"b6cd247f-abb5-4903-8c21-21d8caf8d110\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") "
Nov 01 23:23:45.257483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.257426    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cd247f-abb5-4903-8c21-21d8caf8d110-kube-api-access-j4ksr\" (UniqueName: \"kubernetes.io/projected/b6cd247f-abb5-4903-8c21-21d8caf8d110-kube-api-access-j4ksr\") pod \"b6cd247f-abb5-4903-8c21-21d8caf8d110\" (UID: \"b6cd247f-abb5-4903-8c21-21d8caf8d110\") "
Nov 01 23:23:45.258107 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.258033    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cd247f-abb5-4903-8c21-21d8caf8d110-openshift-service-ca" pod "b6cd247f-abb5-4903-8c21-21d8caf8d110" (UID: "b6cd247f-abb5-4903-8c21-21d8caf8d110"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:23:45.259889 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.259861    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-metrics-certs" pod "b6cd247f-abb5-4903-8c21-21d8caf8d110" (UID: "b6cd247f-abb5-4903-8c21-21d8caf8d110"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:23:45.260200 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.260131    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls" pod "b6cd247f-abb5-4903-8c21-21d8caf8d110" (UID: "b6cd247f-abb5-4903-8c21-21d8caf8d110"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:23:45.260312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.260284    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cd247f-abb5-4903-8c21-21d8caf8d110-kube-api-access-j4ksr" pod "b6cd247f-abb5-4903-8c21-21d8caf8d110" (UID: "b6cd247f-abb5-4903-8c21-21d8caf8d110"). InnerVolumeSpecName "kube-api-access-j4ksr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:23:45.337217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.337188    4837 generic.go:356] "Generic (PLEG): container finished" podID="b6cd247f-abb5-4903-8c21-21d8caf8d110" containerID="d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b" exitCode=0
Nov 01 23:23:45.337356 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.337240    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"
Nov 01 23:23:45.337356 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.337267    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm" event={"ID":"b6cd247f-abb5-4903-8c21-21d8caf8d110","Type":"ContainerDied","Data":"d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b"}
Nov 01 23:23:45.337356 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.337325    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm" event={"ID":"b6cd247f-abb5-4903-8c21-21d8caf8d110","Type":"ContainerDied","Data":"34cf95d40cfae11228f63715ec19ea4e95f4cd48302bceeac4089f73fdd84cb6"}
Nov 01 23:23:45.337493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.337356    4837 scope.go:122] "RemoveContainer" containerID="d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b"
Nov 01 23:23:45.346951 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.346936    4837 scope.go:122] "RemoveContainer" containerID="d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b"
Nov 01 23:23:45.347232 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:23:45.347214    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b\": container with ID starting with d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b not found: ID does not exist" containerID="d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b"
Nov 01 23:23:45.347285 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.347242    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b"} err="failed to get container status \"d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b\": rpc error: code = NotFound desc = could not find container \"d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b\": container with ID starting with d6dfa11e37827525cc41fca161221504ca5702c0f20e7e01db1aa5fe7c1b800b not found: ID does not exist"
Nov 01 23:23:45.358760 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.358736    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cd247f-abb5-4903-8c21-21d8caf8d110-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:45.358901 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.358762    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4ksr\" (UniqueName: \"kubernetes.io/projected/b6cd247f-abb5-4903-8c21-21d8caf8d110-kube-api-access-j4ksr\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:45.358901 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.358773    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:45.358901 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.358781    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6cd247f-abb5-4903-8c21-21d8caf8d110-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:45.492413 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.492373    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"]
Nov 01 23:23:45.494547 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:45.494517    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3fc07c-cdfbb7598-tsnpm"]
Nov 01 23:23:46.523447 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:46.523414    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cd247f-abb5-4903-8c21-21d8caf8d110" path="/var/lib/kubelet/pods/b6cd247f-abb5-4903-8c21-21d8caf8d110/volumes"
Nov 01 23:23:48.281405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:48.281363    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w" podUID="9f6cb910-1c84-4b34-828a-04a2716511d8" containerName="payload-pre-processing" containerID="cri-o://d400596fc696fbda163745eecd07b6bd5835d1f6a5d5924b5a4138fac08c0eb6" gracePeriod=25
Nov 01 23:23:48.813406 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:23:48.813371    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:23:49.166500 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:49.166474    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:23:51.042374 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:51.042337    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"]
Nov 01 23:23:51.088883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:51.088845    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9l8pn"]
Nov 01 23:23:52.525371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:52.525333    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448ab2fc-384d-476b-a321-24463fda13c9" path="/var/lib/kubelet/pods/448ab2fc-384d-476b-a321-24463fda13c9/volumes"
Nov 01 23:23:57.822132 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:23:57.822099    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w" podUID="9f6cb910-1c84-4b34-828a-04a2716511d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:24:00.676281 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.676245    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"]
Nov 01 23:24:00.722723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.722691    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6zgnj"]
Nov 01 23:24:00.795293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.795263    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"]
Nov 01 23:24:00.795695 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.795683    4837 cpu_manager.go:414] "containerMap: removing container" podUID="b6cd247f-abb5-4903-8c21-21d8caf8d110" containerName="maas-api"
Nov 01 23:24:00.795740 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.795696    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cd247f-abb5-4903-8c21-21d8caf8d110" containerName="maas-api"
Nov 01 23:24:00.795788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.795780    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cd247f-abb5-4903-8c21-21d8caf8d110" containerName="maas-api"
Nov 01 23:24:00.800525 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.800504    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.809220 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.809196    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.809351 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.809228    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbjcz\" (UniqueName: \"kubernetes.io/projected/329c8875-5dc4-41e9-8fa5-44096cecfc8c-kube-api-access-bbjcz\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.809351 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.809339    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.851444 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.851413    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:24:00.851584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.851496    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:24:00.851584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.851495    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:24:00.851720 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.851591    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:24:00.854173 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.854146    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"]
Nov 01 23:24:00.910778 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.910748    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.910999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.910818    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.910999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.910853    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbjcz\" (UniqueName: \"kubernetes.io/projected/329c8875-5dc4-41e9-8fa5-44096cecfc8c-kube-api-access-bbjcz\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.911668 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.911641    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.913467 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.913433    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:00.965080 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:00.964986    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbjcz\" (UniqueName: \"kubernetes.io/projected/329c8875-5dc4-41e9-8fa5-44096cecfc8c-kube-api-access-bbjcz\") pod \"maas-api-revoke-sub-3d4156f1-glxzs\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:01.114211 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:01.114171    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:01.480577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:01.480548    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"]
Nov 01 23:24:01.482279 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:24:01.482245    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329c8875_5dc4_41e9_8fa5_44096cecfc8c.slice/crio-bca55f53dc3ef9a193c3af1efb601868b99858336964c802105fb39ad504f231 WatchSource:0}: Error finding container bca55f53dc3ef9a193c3af1efb601868b99858336964c802105fb39ad504f231: Status 404 returned error can't find the container with id bca55f53dc3ef9a193c3af1efb601868b99858336964c802105fb39ad504f231
Nov 01 23:24:02.432199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:02.432163    4837 generic.go:356] "Generic (PLEG): container finished" podID="329c8875-5dc4-41e9-8fa5-44096cecfc8c" containerID="3ab9700d0cb3e3b682c9315c3d41936ceb59bb7c7fe398be1cecc3c12a7f570d" exitCode=0
Nov 01 23:24:02.432581 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:02.432230    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs" event={"ID":"329c8875-5dc4-41e9-8fa5-44096cecfc8c","Type":"ContainerDied","Data":"3ab9700d0cb3e3b682c9315c3d41936ceb59bb7c7fe398be1cecc3c12a7f570d"}
Nov 01 23:24:02.432581 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:02.432282    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs" event={"ID":"329c8875-5dc4-41e9-8fa5-44096cecfc8c","Type":"ContainerStarted","Data":"bca55f53dc3ef9a193c3af1efb601868b99858336964c802105fb39ad504f231"}
Nov 01 23:24:02.523691 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:02.523651    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff37b204-f780-49ba-9605-d12d65c58293" path="/var/lib/kubelet/pods/ff37b204-f780-49ba-9605-d12d65c58293/volumes"
Nov 01 23:24:03.582702 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.582676    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:03.637439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.637406    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-xks-ca\") pod \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") "
Nov 01 23:24:03.637584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.637446    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-openshift-ca\") pod \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") "
Nov 01 23:24:03.637584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.637503    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329c8875-5dc4-41e9-8fa5-44096cecfc8c-kube-api-access-bbjcz\" (UniqueName: \"kubernetes.io/projected/329c8875-5dc4-41e9-8fa5-44096cecfc8c-kube-api-access-bbjcz\") pod \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\" (UID: \"329c8875-5dc4-41e9-8fa5-44096cecfc8c\") "
Nov 01 23:24:03.637869 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.637845    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-openshift-ca" pod "329c8875-5dc4-41e9-8fa5-44096cecfc8c" (UID: "329c8875-5dc4-41e9-8fa5-44096cecfc8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:24:03.639687 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.639664    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329c8875-5dc4-41e9-8fa5-44096cecfc8c-kube-api-access-bbjcz" pod "329c8875-5dc4-41e9-8fa5-44096cecfc8c" (UID: "329c8875-5dc4-41e9-8fa5-44096cecfc8c"). InnerVolumeSpecName "kube-api-access-bbjcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:24:03.639770 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.639733    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-xks-ca" pod "329c8875-5dc4-41e9-8fa5-44096cecfc8c" (UID: "329c8875-5dc4-41e9-8fa5-44096cecfc8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:24:03.739253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.739165    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:03.739253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.739206    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c8875-5dc4-41e9-8fa5-44096cecfc8c-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:03.739253 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:03.739218    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbjcz\" (UniqueName: \"kubernetes.io/projected/329c8875-5dc4-41e9-8fa5-44096cecfc8c-kube-api-access-bbjcz\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.443748 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:04.443723    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"
Nov 01 23:24:04.443946 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:04.443758    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs" event={"ID":"329c8875-5dc4-41e9-8fa5-44096cecfc8c","Type":"ContainerDied","Data":"bca55f53dc3ef9a193c3af1efb601868b99858336964c802105fb39ad504f231"}
Nov 01 23:24:04.443946 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:04.443827    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca55f53dc3ef9a193c3af1efb601868b99858336964c802105fb39ad504f231"
Nov 01 23:24:07.821986 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:07.821952    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w" podUID="9f6cb910-1c84-4b34-828a-04a2716511d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:24:08.467611 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.467577    4837 generic.go:356] "Generic (PLEG): container finished" podID="9f6cb910-1c84-4b34-828a-04a2716511d8" containerID="d400596fc696fbda163745eecd07b6bd5835d1f6a5d5924b5a4138fac08c0eb6" exitCode=0
Nov 01 23:24:08.467796 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.467657    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w" event={"ID":"9f6cb910-1c84-4b34-828a-04a2716511d8","Type":"ContainerDied","Data":"d400596fc696fbda163745eecd07b6bd5835d1f6a5d5924b5a4138fac08c0eb6"}
Nov 01 23:24:08.538631 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.538573    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:24:08.580691 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.580663    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6cb910-1c84-4b34-828a-04a2716511d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6cb910-1c84-4b34-828a-04a2716511d8-plugins-config-volume\") pod \"9f6cb910-1c84-4b34-828a-04a2716511d8\" (UID: \"9f6cb910-1c84-4b34-828a-04a2716511d8\") "
Nov 01 23:24:08.580864 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.580721    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6cb910-1c84-4b34-828a-04a2716511d8-kube-api-access-slndp\" (UniqueName: \"kubernetes.io/projected/9f6cb910-1c84-4b34-828a-04a2716511d8-kube-api-access-slndp\") pod \"9f6cb910-1c84-4b34-828a-04a2716511d8\" (UID: \"9f6cb910-1c84-4b34-828a-04a2716511d8\") "
Nov 01 23:24:08.581111 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.581086    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6cb910-1c84-4b34-828a-04a2716511d8-plugins-config-volume" pod "9f6cb910-1c84-4b34-828a-04a2716511d8" (UID: "9f6cb910-1c84-4b34-828a-04a2716511d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:24:08.582805 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.582782    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6cb910-1c84-4b34-828a-04a2716511d8-kube-api-access-slndp" pod "9f6cb910-1c84-4b34-828a-04a2716511d8" (UID: "9f6cb910-1c84-4b34-828a-04a2716511d8"). InnerVolumeSpecName "kube-api-access-slndp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:24:08.681594 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.681557    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f6cb910-1c84-4b34-828a-04a2716511d8-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:08.681594 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:08.681586    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slndp\" (UniqueName: \"kubernetes.io/projected/9f6cb910-1c84-4b34-828a-04a2716511d8-kube-api-access-slndp\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:09.474012 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:09.473979    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"
Nov 01 23:24:09.474441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:09.474011    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w" event={"ID":"9f6cb910-1c84-4b34-828a-04a2716511d8","Type":"ContainerDied","Data":"24894389973d373cd6d132e1d9414546a4aad2c6f6648a33176f5fc14ce92e82"}
Nov 01 23:24:09.474441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:09.474096    4837 scope.go:122] "RemoveContainer" containerID="d400596fc696fbda163745eecd07b6bd5835d1f6a5d5924b5a4138fac08c0eb6"
Nov 01 23:24:09.627678 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:09.627644    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"]
Nov 01 23:24:09.631838 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:09.631816    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3fc07c-755d6dk94w"]
Nov 01 23:24:10.523202 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:10.523169    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6cb910-1c84-4b34-828a-04a2716511d8" path="/var/lib/kubelet/pods/9f6cb910-1c84-4b34-828a-04a2716511d8/volumes"
Nov 01 23:24:18.812006 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:24:18.811936    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:24:19.043992 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:19.043956    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"]
Nov 01 23:24:19.090293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:19.090201    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4vhr4"]
Nov 01 23:24:20.524728 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:20.524700    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28372765-75c6-4a89-85cf-d8f63a40fc2e" path="/var/lib/kubelet/pods/28372765-75c6-4a89-85cf-d8f63a40fc2e/volumes"
Nov 01 23:24:40.455632 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:40.455596    4837 scope.go:122] "RemoveContainer" containerID="31d93b755e5132c860e3a12712d3230dae057e5f15799d67687012d75cf4f7d9"
Nov 01 23:24:40.464416 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:40.464393    4837 scope.go:122] "RemoveContainer" containerID="840dee8ecd6a34301c934c676434b64248ae8838a9d1060026e0a1d9306cde48"
Nov 01 23:24:48.812255 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:24:48.812216    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:24:50.045559 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:50.045521    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"]
Nov 01 23:24:50.091847 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:50.091811    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n99mz"]
Nov 01 23:24:50.523518 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:24:50.523486    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dabc6f2-fd11-4ac5-90fc-410372242a08" path="/var/lib/kubelet/pods/8dabc6f2-fd11-4ac5-90fc-410372242a08/volumes"
Nov 01 23:25:14.041973 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:14.041938    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"]
Nov 01 23:25:14.088368 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:14.088335    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-xzdcg"]
Nov 01 23:25:14.523818 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:14.523787    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d71455-c93a-4c15-bd57-d46e600703fc" path="/var/lib/kubelet/pods/b4d71455-c93a-4c15-bd57-d46e600703fc/volumes"
Nov 01 23:25:18.812497 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:25:18.812461    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:25:40.499428 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:40.499396    4837 scope.go:122] "RemoveContainer" containerID="ea1ddf344b07c6ced34f01774b4cad8cad1dc4c8f16ceb0d6cd65a8537c8d53a"
Nov 01 23:25:40.511306 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:40.511284    4837 scope.go:122] "RemoveContainer" containerID="3bacdc76885f21b199119e6623bf47baf8a754783ffe93ec941bd831e7f2c1bc"
Nov 01 23:25:44.565089 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:44.565054    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:25:48.811713 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:25:48.811633    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:25:48.875412 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.875374    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"]
Nov 01 23:25:48.875982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.875962    4837 cpu_manager.go:414] "containerMap: removing container" podUID="329c8875-5dc4-41e9-8fa5-44096cecfc8c" containerName="revoke-keys"
Nov 01 23:25:48.875982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.875980    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329c8875-5dc4-41e9-8fa5-44096cecfc8c" containerName="revoke-keys"
Nov 01 23:25:48.876213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.876011    4837 cpu_manager.go:414] "containerMap: removing container" podUID="9f6cb910-1c84-4b34-828a-04a2716511d8" containerName="payload-pre-processing"
Nov 01 23:25:48.876213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.876025    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6cb910-1c84-4b34-828a-04a2716511d8" containerName="payload-pre-processing"
Nov 01 23:25:48.876213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.876164    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="329c8875-5dc4-41e9-8fa5-44096cecfc8c" containerName="revoke-keys"
Nov 01 23:25:48.876213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.876178    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6cb910-1c84-4b34-828a-04a2716511d8" containerName="payload-pre-processing"
Nov 01 23:25:48.879598 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.879580    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:48.930123 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.930093    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:25:48.976313 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.976282    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:25:48.976507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.976492    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:25:48.976576 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.976566    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:25:48.978596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:48.978577    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"]
Nov 01 23:25:49.004497 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.004470    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.004617 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.004520    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn748\" (UniqueName: \"kubernetes.io/projected/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-kube-api-access-cn748\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.004676 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.004654    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.105511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.105441    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.105511 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.105500    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.105701 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.105540    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn748\" (UniqueName: \"kubernetes.io/projected/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-kube-api-access-cn748\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.106124 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.106101    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.107788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.107768    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.159437 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.159413    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn748\" (UniqueName: \"kubernetes.io/projected/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-kube-api-access-cn748\") pod \"maas-api-revoke-sub-d576e68b-jxnxr\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.192377 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.192350    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:49.555512 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:49.555483    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"]
Nov 01 23:25:49.557668 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:25:49.557643    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfd9fce_f33e_4a43_a9a3_c6b64b81b7b7.slice/crio-0543771a11388f7174e6b3cf53db81eb3b2ce81c4e753db9d58fbf136ea32abc WatchSource:0}: Error finding container 0543771a11388f7174e6b3cf53db81eb3b2ce81c4e753db9d58fbf136ea32abc: Status 404 returned error can't find the container with id 0543771a11388f7174e6b3cf53db81eb3b2ce81c4e753db9d58fbf136ea32abc
Nov 01 23:25:50.034565 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:50.034534    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr" event={"ID":"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7","Type":"ContainerStarted","Data":"7abc8d6131f8b32c5e3719a7b600ea773cbe2c2bd81df1243e3e0eee35023df1"}
Nov 01 23:25:50.034987 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:50.034583    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr" event={"ID":"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7","Type":"ContainerStarted","Data":"0543771a11388f7174e6b3cf53db81eb3b2ce81c4e753db9d58fbf136ea32abc"}
Nov 01 23:25:50.142540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:50.142447    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr" podStartSLOduration=2.142433703 podStartE2EDuration="2.142433703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:25:50.140980236 +0000 UTC m=+3731.730179299" watchObservedRunningTime="2026-11-01 23:25:50.142433703 +0000 UTC m=+3731.731632764"
Nov 01 23:25:51.040798 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:51.040764    4837 generic.go:356] "Generic (PLEG): container finished" podID="acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7" containerID="7abc8d6131f8b32c5e3719a7b600ea773cbe2c2bd81df1243e3e0eee35023df1" exitCode=0
Nov 01 23:25:51.041213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:51.040853    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr" event={"ID":"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7","Type":"ContainerDied","Data":"7abc8d6131f8b32c5e3719a7b600ea773cbe2c2bd81df1243e3e0eee35023df1"}
Nov 01 23:25:52.175717 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.175692    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:52.338382 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.338307    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-openshift-ca\") pod \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") "
Nov 01 23:25:52.338382 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.338360    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-xks-ca\") pod \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") "
Nov 01 23:25:52.338608 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.338388    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-kube-api-access-cn748\" (UniqueName: \"kubernetes.io/projected/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-kube-api-access-cn748\") pod \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\" (UID: \"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7\") "
Nov 01 23:25:52.338693 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.338668    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-openshift-ca" pod "acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7" (UID: "acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:25:52.340461 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.340436    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-xks-ca" pod "acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7" (UID: "acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:25:52.340569 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.340493    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-kube-api-access-cn748" pod "acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7" (UID: "acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7"). InnerVolumeSpecName "kube-api-access-cn748". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:25:52.439329 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.439291    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:25:52.439329 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.439324    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn748\" (UniqueName: \"kubernetes.io/projected/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-kube-api-access-cn748\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:25:52.439329 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:52.439335    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:25:53.051872 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:53.051847    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"
Nov 01 23:25:53.052080 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:53.051848    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr" event={"ID":"acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7","Type":"ContainerDied","Data":"0543771a11388f7174e6b3cf53db81eb3b2ce81c4e753db9d58fbf136ea32abc"}
Nov 01 23:25:53.052080 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:25:53.051952    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0543771a11388f7174e6b3cf53db81eb3b2ce81c4e753db9d58fbf136ea32abc"
Nov 01 23:26:01.460761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:01.460728    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:26:08.758066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:08.752215    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:26:08.758066 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:08.752293    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:26:08.791069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:08.790138    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:26:08.791069 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:08.790244    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:26:10.549846 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:10.549817    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:26:17.044057 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.044005    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"]
Nov 01 23:26:17.090577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.090533    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t8jpz"]
Nov 01 23:26:17.872658 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.872627    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"]
Nov 01 23:26:17.873113 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.873097    4837 cpu_manager.go:414] "containerMap: removing container" podUID="acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7" containerName="revoke-keys"
Nov 01 23:26:17.873167 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.873114    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7" containerName="revoke-keys"
Nov 01 23:26:17.873224 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.873214    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7" containerName="revoke-keys"
Nov 01 23:26:17.876228 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.876208    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:17.924220 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.924193    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:17.924358 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.924246    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6w2x\" (UniqueName: \"kubernetes.io/projected/ccb259dc-57fa-4720-81ff-4a87fffafb01-kube-api-access-t6w2x\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:17.924358 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.924339    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:17.925224 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.925205    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:26:17.971777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.971752    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:26:17.971777 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.971753    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:26:17.971988 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.971826    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:26:17.973651 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:17.973628    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"]
Nov 01 23:26:18.025432 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.025402    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:18.025571 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.025539    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:18.025641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.025586    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6w2x\" (UniqueName: \"kubernetes.io/projected/ccb259dc-57fa-4720-81ff-4a87fffafb01-kube-api-access-t6w2x\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:18.026214 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.026196    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:18.027703 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.027674    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:18.079637 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.079605    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6w2x\" (UniqueName: \"kubernetes.io/projected/ccb259dc-57fa-4720-81ff-4a87fffafb01-kube-api-access-t6w2x\") pod \"maas-api-revoke-sub-d2bc93bc-85tw2\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:18.190270 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.190241    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:18.522982 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.522897    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be3de8a-aedc-45fe-b4ed-edc9243b525e" path="/var/lib/kubelet/pods/8be3de8a-aedc-45fe-b4ed-edc9243b525e/volumes"
Nov 01 23:26:18.555537 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:18.555509    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"]
Nov 01 23:26:18.832027 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:26:18.831988    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:26:19.184998 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:19.184960    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2" event={"ID":"ccb259dc-57fa-4720-81ff-4a87fffafb01","Type":"ContainerStarted","Data":"a86576a9047b7f60ed44176e2ef43f45ce30b999900908eab6bfa34faf5b8430"}
Nov 01 23:26:19.185401 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:19.185071    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2" event={"ID":"ccb259dc-57fa-4720-81ff-4a87fffafb01","Type":"ContainerStarted","Data":"c4f260e14f030142aadee3bc774592e02ec1967886296dac39b0ebf956614b58"}
Nov 01 23:26:19.293698 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:19.293642    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2" podStartSLOduration=2.293628637 podStartE2EDuration="2.293628637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:26:19.292705297 +0000 UTC m=+3760.881904358" watchObservedRunningTime="2026-11-01 23:26:19.293628637 +0000 UTC m=+3760.882827698"
Nov 01 23:26:20.191261 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:20.191226    4837 generic.go:356] "Generic (PLEG): container finished" podID="ccb259dc-57fa-4720-81ff-4a87fffafb01" containerID="a86576a9047b7f60ed44176e2ef43f45ce30b999900908eab6bfa34faf5b8430" exitCode=0
Nov 01 23:26:20.191649 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:20.191312    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2" event={"ID":"ccb259dc-57fa-4720-81ff-4a87fffafb01","Type":"ContainerDied","Data":"a86576a9047b7f60ed44176e2ef43f45ce30b999900908eab6bfa34faf5b8430"}
Nov 01 23:26:21.337375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.337350    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:21.458160 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.458085    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-xks-ca\") pod \"ccb259dc-57fa-4720-81ff-4a87fffafb01\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") "
Nov 01 23:26:21.458160 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.458146    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-openshift-ca\") pod \"ccb259dc-57fa-4720-81ff-4a87fffafb01\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") "
Nov 01 23:26:21.458412 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.458181    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb259dc-57fa-4720-81ff-4a87fffafb01-kube-api-access-t6w2x\" (UniqueName: \"kubernetes.io/projected/ccb259dc-57fa-4720-81ff-4a87fffafb01-kube-api-access-t6w2x\") pod \"ccb259dc-57fa-4720-81ff-4a87fffafb01\" (UID: \"ccb259dc-57fa-4720-81ff-4a87fffafb01\") "
Nov 01 23:26:21.459004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.458965    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-openshift-ca" pod "ccb259dc-57fa-4720-81ff-4a87fffafb01" (UID: "ccb259dc-57fa-4720-81ff-4a87fffafb01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:26:21.460512 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.460487    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-xks-ca" pod "ccb259dc-57fa-4720-81ff-4a87fffafb01" (UID: "ccb259dc-57fa-4720-81ff-4a87fffafb01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:26:21.460946 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.460922    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb259dc-57fa-4720-81ff-4a87fffafb01-kube-api-access-t6w2x" pod "ccb259dc-57fa-4720-81ff-4a87fffafb01" (UID: "ccb259dc-57fa-4720-81ff-4a87fffafb01"). InnerVolumeSpecName "kube-api-access-t6w2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:26:21.559255 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.559225    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:26:21.559255 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.559255    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb259dc-57fa-4720-81ff-4a87fffafb01-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:26:21.559472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:21.559270    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6w2x\" (UniqueName: \"kubernetes.io/projected/ccb259dc-57fa-4720-81ff-4a87fffafb01-kube-api-access-t6w2x\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:26:22.203110 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:22.203078    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"
Nov 01 23:26:22.203293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:22.203066    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2" event={"ID":"ccb259dc-57fa-4720-81ff-4a87fffafb01","Type":"ContainerDied","Data":"c4f260e14f030142aadee3bc774592e02ec1967886296dac39b0ebf956614b58"}
Nov 01 23:26:22.203293 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:22.203200    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4f260e14f030142aadee3bc774592e02ec1967886296dac39b0ebf956614b58"
Nov 01 23:26:28.043059 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:28.043010    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"]
Nov 01 23:26:28.089362 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:28.089319    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fkstw"]
Nov 01 23:26:28.523056 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:28.523010    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88194a3-dcd1-4655-a5af-dba65f7f7dff" path="/var/lib/kubelet/pods/d88194a3-dcd1-4655-a5af-dba65f7f7dff/volumes"
Nov 01 23:26:30.676841 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:30.676795    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:26:38.798154 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.798121    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"]
Nov 01 23:26:38.798554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.798540    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ccb259dc-57fa-4720-81ff-4a87fffafb01" containerName="revoke-keys"
Nov 01 23:26:38.798597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.798554    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb259dc-57fa-4720-81ff-4a87fffafb01" containerName="revoke-keys"
Nov 01 23:26:38.798638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.798626    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb259dc-57fa-4720-81ff-4a87fffafb01" containerName="revoke-keys"
Nov 01 23:26:38.803090 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.803068    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:38.901098 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.900519    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0cff29\""
Nov 01 23:26:38.901098 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.900692    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-cb26499d-dockercfg-b8fsq\""
Nov 01 23:26:38.905249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.905219    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"]
Nov 01 23:26:38.936355 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.936326    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5bcj\" (UniqueName: \"kubernetes.io/projected/1815a9aa-0f26-466f-869d-aa250f16b252-kube-api-access-t5bcj\") pod \"payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8\" (UID: \"1815a9aa-0f26-466f-869d-aa250f16b252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:38.936521 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:38.936379    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1815a9aa-0f26-466f-869d-aa250f16b252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8\" (UID: \"1815a9aa-0f26-466f-869d-aa250f16b252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:39.037406 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:39.037377    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5bcj\" (UniqueName: \"kubernetes.io/projected/1815a9aa-0f26-466f-869d-aa250f16b252-kube-api-access-t5bcj\") pod \"payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8\" (UID: \"1815a9aa-0f26-466f-869d-aa250f16b252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:39.037577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:39.037431    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1815a9aa-0f26-466f-869d-aa250f16b252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8\" (UID: \"1815a9aa-0f26-466f-869d-aa250f16b252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:39.038173 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:39.038147    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1815a9aa-0f26-466f-869d-aa250f16b252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8\" (UID: \"1815a9aa-0f26-466f-869d-aa250f16b252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:39.091322 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:39.091255    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5bcj\" (UniqueName: \"kubernetes.io/projected/1815a9aa-0f26-466f-869d-aa250f16b252-kube-api-access-t5bcj\") pod \"payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8\" (UID: \"1815a9aa-0f26-466f-869d-aa250f16b252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:39.123054 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:39.123017    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:39.486100 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:39.486067    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"]
Nov 01 23:26:39.487484 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:26:39.487456    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1815a9aa_0f26_466f_869d_aa250f16b252.slice/crio-60d006da1db00375528750fd31fef38e91790f78e0d62af79e343fdef7ee8097 WatchSource:0}: Error finding container 60d006da1db00375528750fd31fef38e91790f78e0d62af79e343fdef7ee8097: Status 404 returned error can't find the container with id 60d006da1db00375528750fd31fef38e91790f78e0d62af79e343fdef7ee8097
Nov 01 23:26:40.294597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:40.294566    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8" event={"ID":"1815a9aa-0f26-466f-869d-aa250f16b252","Type":"ContainerStarted","Data":"bbfe0310b251183f7c8741793ff5ca06ed26c62127383c484a3c747f66733d93"}
Nov 01 23:26:40.295063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:40.294623    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8" event={"ID":"1815a9aa-0f26-466f-869d-aa250f16b252","Type":"ContainerStarted","Data":"60d006da1db00375528750fd31fef38e91790f78e0d62af79e343fdef7ee8097"}
Nov 01 23:26:40.295063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:40.294681    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:26:40.404072 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:40.403995    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8" podStartSLOduration=2.403981966 podStartE2EDuration="2.403981966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:26:40.402848201 +0000 UTC m=+3781.992047261" watchObservedRunningTime="2026-11-01 23:26:40.403981966 +0000 UTC m=+3781.993181026"
Nov 01 23:26:40.545290 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:40.542973    4837 scope.go:122] "RemoveContainer" containerID="7e74bfcf2d5da6a29dea5963bb60f7bc1e00fcf0746f0c08f763e41c4a81f0fc"
Nov 01 23:26:40.555438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:40.555413    4837 scope.go:122] "RemoveContainer" containerID="736cb062350566a876c36d2a88eaad4f16efdbe9efc1ca1279b05a1100282414"
Nov 01 23:26:40.563391 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:40.563361    4837 scope.go:122] "RemoveContainer" containerID="344590d2818547cbd285a09249b9b5fc7328d4899c2c3b9c86df811a1f57ac01"
Nov 01 23:26:40.571189 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:40.571172    4837 scope.go:122] "RemoveContainer" containerID="e3f257ecc7bfc1323ca8084f08edfc87c5a2a2e261c2ebcdf0596f484daff82f"
Nov 01 23:26:41.200869 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.200824    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"]
Nov 01 23:26:41.206182 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.206158    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.304055 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.304013    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 23:26:41.304055 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.304010    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-0cff29-serving-cert\""
Nov 01 23:26:41.304475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.304060    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:26:41.304475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.304086    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:26:41.304475 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.304190    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xp822\""
Nov 01 23:26:41.306194 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.306175    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"]
Nov 01 23:26:41.362184 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.362149    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8c391-51a8-444c-8b41-ff880e58b2d1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.362373 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.362255    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.362373 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.362281    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ng9f\" (UniqueName: \"kubernetes.io/projected/3ca8c391-51a8-444c-8b41-ff880e58b2d1-kube-api-access-7ng9f\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.362373 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.362307    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.467080 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.466979    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.467371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.467342    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ng9f\" (UniqueName: \"kubernetes.io/projected/3ca8c391-51a8-444c-8b41-ff880e58b2d1-kube-api-access-7ng9f\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.467371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.467387    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.467630 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.467528    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8c391-51a8-444c-8b41-ff880e58b2d1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.468360 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.468333    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8c391-51a8-444c-8b41-ff880e58b2d1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.472634 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.472608    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.477735 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.477709    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.521402 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.521375    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ng9f\" (UniqueName: \"kubernetes.io/projected/3ca8c391-51a8-444c-8b41-ff880e58b2d1-kube-api-access-7ng9f\") pod \"maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:41.819557 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:41.819475    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:42.187233 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:42.187210    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"]
Nov 01 23:26:42.188880 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:26:42.188841    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca8c391_51a8_444c_8b41_ff880e58b2d1.slice/crio-25db0c1de489fd8f6a389de0fcd13acfef0b5a78bcbe10e2dd035e1445d71f9e WatchSource:0}: Error finding container 25db0c1de489fd8f6a389de0fcd13acfef0b5a78bcbe10e2dd035e1445d71f9e: Status 404 returned error can't find the container with id 25db0c1de489fd8f6a389de0fcd13acfef0b5a78bcbe10e2dd035e1445d71f9e
Nov 01 23:26:42.309191 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:42.309156    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp" event={"ID":"3ca8c391-51a8-444c-8b41-ff880e58b2d1","Type":"ContainerStarted","Data":"25db0c1de489fd8f6a389de0fcd13acfef0b5a78bcbe10e2dd035e1445d71f9e"}
Nov 01 23:26:43.315680 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:43.315638    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp" event={"ID":"3ca8c391-51a8-444c-8b41-ff880e58b2d1","Type":"ContainerStarted","Data":"5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a"}
Nov 01 23:26:43.316097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:43.315797    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:43.425725 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:43.425673    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp" podStartSLOduration=1.494945694 podStartE2EDuration="2.425657568s" totalImagesPullingTime="930.711874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:26:43.423597924 +0000 UTC m=+3785.012796984" watchObservedRunningTime="2026-11-01 23:26:43.425657568 +0000 UTC m=+3785.014856630"
Nov 01 23:26:48.843664 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:26:48.843629    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:26:48.989027 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:48.988991    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:26:49.326593 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:49.326567    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:26:51.308096 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:26:51.308063    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:27:07.367620 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:27:07.367583    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:27:18.812378 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:27:18.812287    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:27:33.191883 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:27:33.191847    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:27:40.600862 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:27:40.600823    4837 scope.go:122] "RemoveContainer" containerID="7603d024e10d9fb3827b6cc1054583395dd4ff706b313b4cd4820d97cdf7d85f"
Nov 01 23:27:40.609550 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:27:40.609532    4837 scope.go:122] "RemoveContainer" containerID="a7588d9ba8a2b8cb5d77b9bf2c7f0a10c186bfe13d6a478d006fa05304f71100"
Nov 01 23:27:48.811557 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:27:48.811518    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:28:01.388661 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:01.388623    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:28:11.735249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:11.735216    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"]
Nov 01 23:28:16.736218 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:16.736166    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8" podUID="1815a9aa-0f26-466f-869d-aa250f16b252" containerName="payload-pre-processing" containerID="cri-o://bbfe0310b251183f7c8741793ff5ca06ed26c62127383c484a3c747f66733d93" gracePeriod=25
Nov 01 23:28:17.788371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:17.788337    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"]
Nov 01 23:28:17.788921 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:17.788537    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp" podUID="3ca8c391-51a8-444c-8b41-ff880e58b2d1" containerName="maas-api" containerID="cri-o://5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a" gracePeriod=30
Nov 01 23:28:18.812963 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:28:18.812925    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:28:19.049950 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.049923    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:28:19.202416 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.202386    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-maas-api-tls\") pod \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") "
Nov 01 23:28:19.202599 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.202456    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca8c391-51a8-444c-8b41-ff880e58b2d1-kube-api-access-7ng9f\" (UniqueName: \"kubernetes.io/projected/3ca8c391-51a8-444c-8b41-ff880e58b2d1-kube-api-access-7ng9f\") pod \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") "
Nov 01 23:28:19.202599 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.202508    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca8c391-51a8-444c-8b41-ff880e58b2d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8c391-51a8-444c-8b41-ff880e58b2d1-openshift-service-ca\") pod \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") "
Nov 01 23:28:19.202599 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.202594    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-metrics-certs\") pod \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\" (UID: \"3ca8c391-51a8-444c-8b41-ff880e58b2d1\") "
Nov 01 23:28:19.202922 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.202898    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca8c391-51a8-444c-8b41-ff880e58b2d1-openshift-service-ca" pod "3ca8c391-51a8-444c-8b41-ff880e58b2d1" (UID: "3ca8c391-51a8-444c-8b41-ff880e58b2d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:28:19.204582 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.204559    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-maas-api-tls" pod "3ca8c391-51a8-444c-8b41-ff880e58b2d1" (UID: "3ca8c391-51a8-444c-8b41-ff880e58b2d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:28:19.204664 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.204594    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-metrics-certs" pod "3ca8c391-51a8-444c-8b41-ff880e58b2d1" (UID: "3ca8c391-51a8-444c-8b41-ff880e58b2d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:28:19.204769 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.204752    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca8c391-51a8-444c-8b41-ff880e58b2d1-kube-api-access-7ng9f" pod "3ca8c391-51a8-444c-8b41-ff880e58b2d1" (UID: "3ca8c391-51a8-444c-8b41-ff880e58b2d1"). InnerVolumeSpecName "kube-api-access-7ng9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:28:19.304498 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.304466    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:19.304498 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.304494    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ng9f\" (UniqueName: \"kubernetes.io/projected/3ca8c391-51a8-444c-8b41-ff880e58b2d1-kube-api-access-7ng9f\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:19.304718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.304510    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca8c391-51a8-444c-8b41-ff880e58b2d1-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:19.304718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.304523    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca8c391-51a8-444c-8b41-ff880e58b2d1-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:19.815312 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.815276    4837 generic.go:356] "Generic (PLEG): container finished" podID="3ca8c391-51a8-444c-8b41-ff880e58b2d1" containerID="5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a" exitCode=0
Nov 01 23:28:19.815723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.815326    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"
Nov 01 23:28:19.815723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.815341    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp" event={"ID":"3ca8c391-51a8-444c-8b41-ff880e58b2d1","Type":"ContainerDied","Data":"5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a"}
Nov 01 23:28:19.815723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.815391    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp" event={"ID":"3ca8c391-51a8-444c-8b41-ff880e58b2d1","Type":"ContainerDied","Data":"25db0c1de489fd8f6a389de0fcd13acfef0b5a78bcbe10e2dd035e1445d71f9e"}
Nov 01 23:28:19.815723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.815421    4837 scope.go:122] "RemoveContainer" containerID="5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a"
Nov 01 23:28:19.824846 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.824820    4837 scope.go:122] "RemoveContainer" containerID="5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a"
Nov 01 23:28:19.825118 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:28:19.825097    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a\": container with ID starting with 5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a not found: ID does not exist" containerID="5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a"
Nov 01 23:28:19.825217 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.825123    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a"} err="failed to get container status \"5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a\": rpc error: code = NotFound desc = could not find container \"5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a\": container with ID starting with 5670447e4f93010ff92c88d107554ff748cc3fa192f8e0e2a6fa28fefc588c8a not found: ID does not exist"
Nov 01 23:28:19.968439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:19.968411    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"]
Nov 01 23:28:20.015518 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:20.015487    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0cff29-758cb777c8-m7vvp"]
Nov 01 23:28:20.523499 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:20.523468    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca8c391-51a8-444c-8b41-ff880e58b2d1" path="/var/lib/kubelet/pods/3ca8c391-51a8-444c-8b41-ff880e58b2d1/volumes"
Nov 01 23:28:21.304083 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:21.304025    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8" podUID="1815a9aa-0f26-466f-869d-aa250f16b252" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:28:22.249653 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:22.249613    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:28:31.304564 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:31.304523    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8" podUID="1815a9aa-0f26-466f-869d-aa250f16b252" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:28:36.903175 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:36.903134    4837 generic.go:356] "Generic (PLEG): container finished" podID="1815a9aa-0f26-466f-869d-aa250f16b252" containerID="bbfe0310b251183f7c8741793ff5ca06ed26c62127383c484a3c747f66733d93" exitCode=0
Nov 01 23:28:36.903597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:36.903198    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8" event={"ID":"1815a9aa-0f26-466f-869d-aa250f16b252","Type":"ContainerDied","Data":"bbfe0310b251183f7c8741793ff5ca06ed26c62127383c484a3c747f66733d93"}
Nov 01 23:28:36.996642 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:36.996566    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:28:37.064796 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.064757    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1815a9aa-0f26-466f-869d-aa250f16b252-kube-api-access-t5bcj\" (UniqueName: \"kubernetes.io/projected/1815a9aa-0f26-466f-869d-aa250f16b252-kube-api-access-t5bcj\") pod \"1815a9aa-0f26-466f-869d-aa250f16b252\" (UID: \"1815a9aa-0f26-466f-869d-aa250f16b252\") "
Nov 01 23:28:37.065009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.064813    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1815a9aa-0f26-466f-869d-aa250f16b252-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1815a9aa-0f26-466f-869d-aa250f16b252-plugins-config-volume\") pod \"1815a9aa-0f26-466f-869d-aa250f16b252\" (UID: \"1815a9aa-0f26-466f-869d-aa250f16b252\") "
Nov 01 23:28:37.065357 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.065329    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1815a9aa-0f26-466f-869d-aa250f16b252-plugins-config-volume" pod "1815a9aa-0f26-466f-869d-aa250f16b252" (UID: "1815a9aa-0f26-466f-869d-aa250f16b252"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:28:37.066986 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.066965    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1815a9aa-0f26-466f-869d-aa250f16b252-kube-api-access-t5bcj" pod "1815a9aa-0f26-466f-869d-aa250f16b252" (UID: "1815a9aa-0f26-466f-869d-aa250f16b252"). InnerVolumeSpecName "kube-api-access-t5bcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:28:37.093552 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.093525    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:28:37.166723 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.166690    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5bcj\" (UniqueName: \"kubernetes.io/projected/1815a9aa-0f26-466f-869d-aa250f16b252-kube-api-access-t5bcj\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:37.167005 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.166971    4837 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1815a9aa-0f26-466f-869d-aa250f16b252-plugins-config-volume\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:37.912842 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.912790    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8" event={"ID":"1815a9aa-0f26-466f-869d-aa250f16b252","Type":"ContainerDied","Data":"60d006da1db00375528750fd31fef38e91790f78e0d62af79e343fdef7ee8097"}
Nov 01 23:28:37.913408 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.912906    4837 scope.go:122] "RemoveContainer" containerID="bbfe0310b251183f7c8741793ff5ca06ed26c62127383c484a3c747f66733d93"
Nov 01 23:28:37.913408 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:37.913117    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"
Nov 01 23:28:38.074159 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:38.074122    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"]
Nov 01 23:28:38.117412 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:38.117377    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0cff29-58d577crq8"]
Nov 01 23:28:38.557034 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:38.556641    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1815a9aa-0f26-466f-869d-aa250f16b252" path="/var/lib/kubelet/pods/1815a9aa-0f26-466f-869d-aa250f16b252/volumes"
Nov 01 23:28:39.305064 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:39.304625    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:28:39.376688 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:39.376659    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:28:40.631999 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:28:40.631971    4837 scope.go:122] "RemoveContainer" containerID="09c9f189a5e1409695f74d5a7c588f8874715a8418b926cbd3b895f34e2eb99d"
Nov 01 23:28:48.814618 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:28:48.814535    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:29:04.045100 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:04.045070    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"]
Nov 01 23:29:04.091333 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:04.091302    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-glxzs"]
Nov 01 23:29:04.524147 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:04.524105    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329c8875-5dc4-41e9-8fa5-44096cecfc8c" path="/var/lib/kubelet/pods/329c8875-5dc4-41e9-8fa5-44096cecfc8c/volumes"
Nov 01 23:29:16.015097 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.015062    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"]
Nov 01 23:29:16.015527 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.015505    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1815a9aa-0f26-466f-869d-aa250f16b252" containerName="payload-pre-processing"
Nov 01 23:29:16.015527 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.015516    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1815a9aa-0f26-466f-869d-aa250f16b252" containerName="payload-pre-processing"
Nov 01 23:29:16.015609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.015534    4837 cpu_manager.go:414] "containerMap: removing container" podUID="3ca8c391-51a8-444c-8b41-ff880e58b2d1" containerName="maas-api"
Nov 01 23:29:16.015609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.015540    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca8c391-51a8-444c-8b41-ff880e58b2d1" containerName="maas-api"
Nov 01 23:29:16.015609 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.015605    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1815a9aa-0f26-466f-869d-aa250f16b252" containerName="payload-pre-processing"
Nov 01 23:29:16.015704 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.015615    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca8c391-51a8-444c-8b41-ff880e58b2d1" containerName="maas-api"
Nov 01 23:29:16.018399 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.018375    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.113447 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.113414    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-863b60f0-kserve-self-signed-certs\""
Nov 01 23:29:16.115439 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.115411    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"]
Nov 01 23:29:16.173575 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.173539    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tmp-dir\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.173781 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.173600    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-home\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.173781 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.173752    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdjn\" (UniqueName: \"kubernetes.io/projected/ee0d91ca-5558-4f71-b10f-afa7e73f4387-kube-api-access-srdjn\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.173894 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.173837    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tls-certs\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.173940 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.173899    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-model-cache\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.173979 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.173956    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-dshm\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.274710 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.274621    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tls-certs\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.274710 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.274673    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-model-cache\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.274710 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.274707    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-dshm\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.275009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.274740    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tmp-dir\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.275009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.274767    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-home\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.275009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.274917    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdjn\" (UniqueName: \"kubernetes.io/projected/ee0d91ca-5558-4f71-b10f-afa7e73f4387-kube-api-access-srdjn\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.275261 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.275221    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-model-cache\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.275334 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.275239    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-home\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.275334 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.275293    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tmp-dir\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.277146 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.277116    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-dshm\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.277307 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.277291    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tls-certs\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.328655 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.328605    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdjn\" (UniqueName: \"kubernetes.io/projected/ee0d91ca-5558-4f71-b10f-afa7e73f4387-kube-api-access-srdjn\") pod \"e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") " pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.344494 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.344459    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:16.732753 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:16.732728    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"]
Nov 01 23:29:16.734995 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:29:16.734962    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee0d91ca_5558_4f71_b10f_afa7e73f4387.slice/crio-1f80cf293db515916b1142657626e27f4cdcc71b8ebce69f4de535719d6b7f19 WatchSource:0}: Error finding container 1f80cf293db515916b1142657626e27f4cdcc71b8ebce69f4de535719d6b7f19: Status 404 returned error can't find the container with id 1f80cf293db515916b1142657626e27f4cdcc71b8ebce69f4de535719d6b7f19
Nov 01 23:29:17.169420 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:17.169383    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v" event={"ID":"ee0d91ca-5558-4f71-b10f-afa7e73f4387","Type":"ContainerStarted","Data":"e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3"}
Nov 01 23:29:17.169960 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:17.169432    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v" event={"ID":"ee0d91ca-5558-4f71-b10f-afa7e73f4387","Type":"ContainerStarted","Data":"1f80cf293db515916b1142657626e27f4cdcc71b8ebce69f4de535719d6b7f19"}
Nov 01 23:29:17.279014 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:17.278959    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v" podStartSLOduration=2.278940016 podStartE2EDuration="2.278940016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:29:17.278402519 +0000 UTC m=+3938.867601582" watchObservedRunningTime="2026-11-01 23:29:17.278940016 +0000 UTC m=+3938.868139077"
Nov 01 23:29:17.345641 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:17.345609    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:18.812086 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:29:18.812033    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:29:20.482985 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:20.482952    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:29:23.750067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:23.749199    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:29:23.750067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:23.749304    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:29:23.785344 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:23.785207    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:29:23.785344 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:23.785272    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:29:26.345325 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:26.345287    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:26.357831 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:26.357804    4837 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:27.270785 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:27.270757    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:29:48.813115 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:29:48.813073    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:29:49.576733 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.576695    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"]
Nov 01 23:29:49.580520 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.580500    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.630154 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.630127    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:29:49.630154 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.630141    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:29:49.649686 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.649657    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.649852 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.649692    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9wg\" (UniqueName: \"kubernetes.io/projected/68f459ca-bf39-436c-8420-7a3c08bb46df-kube-api-access-dd9wg\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.649852 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.649719    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.676458 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.676433    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:29:49.676582 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.676470    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:29:49.678577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.678556    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"]
Nov 01 23:29:49.751254 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.751219    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.751448 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.751265    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9wg\" (UniqueName: \"kubernetes.io/projected/68f459ca-bf39-436c-8420-7a3c08bb46df-kube-api-access-dd9wg\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.751448 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.751302    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.751969 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.751948    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.754065 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.754032    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.805661 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.805630    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9wg\" (UniqueName: \"kubernetes.io/projected/68f459ca-bf39-436c-8420-7a3c08bb46df-kube-api-access-dd9wg\") pod \"maas-api-revoke-sub-452b6d7b-gwvcz\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:49.893815 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:49.893741    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:50.256389 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:50.256365    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"]
Nov 01 23:29:50.258018 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:29:50.257992    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f459ca_bf39_436c_8420_7a3c08bb46df.slice/crio-77ef3ee64d770ec66e1157bba8d1197035fd188f8331d7ef40c215136859c0b0 WatchSource:0}: Error finding container 77ef3ee64d770ec66e1157bba8d1197035fd188f8331d7ef40c215136859c0b0: Status 404 returned error can't find the container with id 77ef3ee64d770ec66e1157bba8d1197035fd188f8331d7ef40c215136859c0b0
Nov 01 23:29:50.378913 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:50.378884    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz" event={"ID":"68f459ca-bf39-436c-8420-7a3c08bb46df","Type":"ContainerStarted","Data":"77ef3ee64d770ec66e1157bba8d1197035fd188f8331d7ef40c215136859c0b0"}
Nov 01 23:29:50.566944 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:50.566869    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:29:51.388868 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:51.388820    4837 generic.go:356] "Generic (PLEG): container finished" podID="68f459ca-bf39-436c-8420-7a3c08bb46df" containerID="d7681c44053c39c37c8cde7f21a93d0d5547da5e89fc25f2c920609ecd990fc0" exitCode=0
Nov 01 23:29:51.389395 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:51.388974    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz" event={"ID":"68f459ca-bf39-436c-8420-7a3c08bb46df","Type":"ContainerDied","Data":"d7681c44053c39c37c8cde7f21a93d0d5547da5e89fc25f2c920609ecd990fc0"}
Nov 01 23:29:53.032896 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.032866    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:53.137385 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.137350    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-openshift-ca\") pod \"68f459ca-bf39-436c-8420-7a3c08bb46df\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") "
Nov 01 23:29:53.137685 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.137657    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-xks-ca\") pod \"68f459ca-bf39-436c-8420-7a3c08bb46df\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") "
Nov 01 23:29:53.137967 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.137946    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f459ca-bf39-436c-8420-7a3c08bb46df-kube-api-access-dd9wg\" (UniqueName: \"kubernetes.io/projected/68f459ca-bf39-436c-8420-7a3c08bb46df-kube-api-access-dd9wg\") pod \"68f459ca-bf39-436c-8420-7a3c08bb46df\" (UID: \"68f459ca-bf39-436c-8420-7a3c08bb46df\") "
Nov 01 23:29:53.138457 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.138431    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-openshift-ca" pod "68f459ca-bf39-436c-8420-7a3c08bb46df" (UID: "68f459ca-bf39-436c-8420-7a3c08bb46df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:29:53.158063 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.154735    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f459ca-bf39-436c-8420-7a3c08bb46df-kube-api-access-dd9wg" pod "68f459ca-bf39-436c-8420-7a3c08bb46df" (UID: "68f459ca-bf39-436c-8420-7a3c08bb46df"). InnerVolumeSpecName "kube-api-access-dd9wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:29:53.158628 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.158604    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-xks-ca" pod "68f459ca-bf39-436c-8420-7a3c08bb46df" (UID: "68f459ca-bf39-436c-8420-7a3c08bb46df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:29:53.253008 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.252903    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:29:53.253292 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.253272    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f459ca-bf39-436c-8420-7a3c08bb46df-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:29:53.253432 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.253416    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9wg\" (UniqueName: \"kubernetes.io/projected/68f459ca-bf39-436c-8420-7a3c08bb46df-kube-api-access-dd9wg\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:29:53.479067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.478618    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz" event={"ID":"68f459ca-bf39-436c-8420-7a3c08bb46df","Type":"ContainerDied","Data":"77ef3ee64d770ec66e1157bba8d1197035fd188f8331d7ef40c215136859c0b0"}
Nov 01 23:29:53.479067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.478701    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77ef3ee64d770ec66e1157bba8d1197035fd188f8331d7ef40c215136859c0b0"
Nov 01 23:29:53.479067 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.478853    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"
Nov 01 23:29:53.760068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.759174    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:29:53.760068 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.759290    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:29:53.799134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.797161    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:29:53.799134 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:29:53.797229    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:30:00.152848 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.152807    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"]
Nov 01 23:30:00.153315 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.153299    4837 cpu_manager.go:414] "containerMap: removing container" podUID="68f459ca-bf39-436c-8420-7a3c08bb46df" containerName="revoke-keys"
Nov 01 23:30:00.153362 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.153317    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f459ca-bf39-436c-8420-7a3c08bb46df" containerName="revoke-keys"
Nov 01 23:30:00.153424 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.153414    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f459ca-bf39-436c-8420-7a3c08bb46df" containerName="revoke-keys"
Nov 01 23:30:00.156891 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.156869    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:00.254798 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.254770    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:30:00.254798 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.254771    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:30:00.255009 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.254772    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:30:00.256722 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.256702    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"]
Nov 01 23:30:00.286899 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.286877    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892930-w78l5\" (UID: \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:00.287074 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.286917    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvkc\" (UniqueName: \"kubernetes.io/projected/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-kube-api-access-2jvkc\") pod \"maas-api-key-cleanup-29892930-w78l5\" (UID: \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:00.387944 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.387911    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892930-w78l5\" (UID: \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:00.388135 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.387959    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvkc\" (UniqueName: \"kubernetes.io/projected/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-kube-api-access-2jvkc\") pod \"maas-api-key-cleanup-29892930-w78l5\" (UID: \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:00.388571 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.388552    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892930-w78l5\" (UID: \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:00.441871 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.441826    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvkc\" (UniqueName: \"kubernetes.io/projected/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-kube-api-access-2jvkc\") pod \"maas-api-key-cleanup-29892930-w78l5\" (UID: \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:00.470493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.470470    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:00.836945 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:00.836918    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"]
Nov 01 23:30:00.838058 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:30:00.838011    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed38ab7_4bab_40e9_90d1_8532ad7ae572.slice/crio-b06649e4aa28130af476e3b9c6faef2e71a08429b5215f6c92d1dbe8a0651294 WatchSource:0}: Error finding container b06649e4aa28130af476e3b9c6faef2e71a08429b5215f6c92d1dbe8a0651294: Status 404 returned error can't find the container with id b06649e4aa28130af476e3b9c6faef2e71a08429b5215f6c92d1dbe8a0651294
Nov 01 23:30:01.523318 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:01.523280    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" event={"ID":"1ed38ab7-4bab-40e9-90d1-8532ad7ae572","Type":"ContainerStarted","Data":"97b473ca3757a171061c84773dbd8f8ec082cff278329e056631be36e55adb26"}
Nov 01 23:30:01.523833 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:01.523346    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" event={"ID":"1ed38ab7-4bab-40e9-90d1-8532ad7ae572","Type":"ContainerStarted","Data":"b06649e4aa28130af476e3b9c6faef2e71a08429b5215f6c92d1dbe8a0651294"}
Nov 01 23:30:01.630147 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:01.630094    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" podStartSLOduration=1.6300784560000001 podStartE2EDuration="1.630078456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:01.628259952 +0000 UTC m=+3983.217459032" watchObservedRunningTime="2026-11-01 23:30:01.630078456 +0000 UTC m=+3983.219277517"
Nov 01 23:30:02.529269 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:02.529234    4837 generic.go:356] "Generic (PLEG): container finished" podID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerID="97b473ca3757a171061c84773dbd8f8ec082cff278329e056631be36e55adb26" exitCode=60
Nov 01 23:30:02.529682 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:02.529322    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" event={"ID":"1ed38ab7-4bab-40e9-90d1-8532ad7ae572","Type":"ContainerDied","Data":"97b473ca3757a171061c84773dbd8f8ec082cff278329e056631be36e55adb26"}
Nov 01 23:30:02.529682 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:02.529542    4837 scope.go:122] "RemoveContainer" containerID="97b473ca3757a171061c84773dbd8f8ec082cff278329e056631be36e55adb26"
Nov 01 23:30:03.536248 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:03.536212    4837 generic.go:356] "Generic (PLEG): container finished" podID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerID="b231453858786e9d0d94111b27339e4ea04e5df1dd216e91871a011749be2ed6" exitCode=60
Nov 01 23:30:03.536716 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:03.536256    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" event={"ID":"1ed38ab7-4bab-40e9-90d1-8532ad7ae572","Type":"ContainerDied","Data":"b231453858786e9d0d94111b27339e4ea04e5df1dd216e91871a011749be2ed6"}
Nov 01 23:30:03.536716 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:03.536305    4837 scope.go:122] "RemoveContainer" containerID="97b473ca3757a171061c84773dbd8f8ec082cff278329e056631be36e55adb26"
Nov 01 23:30:03.536716 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:03.536524    4837 scope.go:122] "RemoveContainer" containerID="b231453858786e9d0d94111b27339e4ea04e5df1dd216e91871a011749be2ed6"
Nov 01 23:30:03.536892 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:30:03.536794    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892930-w78l5_odh-ai-gateway-infra(1ed38ab7-4bab-40e9-90d1-8532ad7ae572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572"
Nov 01 23:30:04.542305 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:04.542277    4837 scope.go:122] "RemoveContainer" containerID="b231453858786e9d0d94111b27339e4ea04e5df1dd216e91871a011749be2ed6"
Nov 01 23:30:04.542697 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:30:04.542493    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892930-w78l5_odh-ai-gateway-infra(1ed38ab7-4bab-40e9-90d1-8532ad7ae572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572"
Nov 01 23:30:05.867930 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:05.867891    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:30:07.127562 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:07.127520    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"]
Nov 01 23:30:08.754213 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:08.754151    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:30:08.754911 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:08.754878    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:30:08.786265 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:08.786153    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:30:08.786265 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:08.786206    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:30:15.517718 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:15.517690    4837 scope.go:122] "RemoveContainer" containerID="b231453858786e9d0d94111b27339e4ea04e5df1dd216e91871a011749be2ed6"
Nov 01 23:30:16.652234 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:16.652148    4837 generic.go:356] "Generic (PLEG): container finished" podID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerID="02a72322ce0f7fca6fad83c3d3cbfc9dc4b7e16fd8aebfb5d339f52916699b81" exitCode=60
Nov 01 23:30:16.652714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:16.652226    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" event={"ID":"1ed38ab7-4bab-40e9-90d1-8532ad7ae572","Type":"ContainerDied","Data":"02a72322ce0f7fca6fad83c3d3cbfc9dc4b7e16fd8aebfb5d339f52916699b81"}
Nov 01 23:30:16.652714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:16.652288    4837 scope.go:122] "RemoveContainer" containerID="b231453858786e9d0d94111b27339e4ea04e5df1dd216e91871a011749be2ed6"
Nov 01 23:30:16.652714 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:16.652659    4837 scope.go:122] "RemoveContainer" containerID="02a72322ce0f7fca6fad83c3d3cbfc9dc4b7e16fd8aebfb5d339f52916699b81"
Nov 01 23:30:16.652979 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:30:16.652959    4837 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892930-w78l5_odh-ai-gateway-infra(1ed38ab7-4bab-40e9-90d1-8532ad7ae572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572"
Nov 01 23:30:17.080061 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:17.080017    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:30:17.771998 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:17.771959    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"]
Nov 01 23:30:17.916944 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:17.916920    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:17.995270 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:17.995237    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-maas-api-service-ca\") pod \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\" (UID: \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\") "
Nov 01 23:30:17.995469 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:17.995300    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-kube-api-access-2jvkc\" (UniqueName: \"kubernetes.io/projected/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-kube-api-access-2jvkc\") pod \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\" (UID: \"1ed38ab7-4bab-40e9-90d1-8532ad7ae572\") "
Nov 01 23:30:17.995696 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:17.995672    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-maas-api-service-ca" pod "1ed38ab7-4bab-40e9-90d1-8532ad7ae572" (UID: "1ed38ab7-4bab-40e9-90d1-8532ad7ae572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:30:17.997491 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:17.997473    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-kube-api-access-2jvkc" pod "1ed38ab7-4bab-40e9-90d1-8532ad7ae572" (UID: "1ed38ab7-4bab-40e9-90d1-8532ad7ae572"). InnerVolumeSpecName "kube-api-access-2jvkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:30:18.096386 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:18.096310    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-maas-api-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:18.096386 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:18.096339    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvkc\" (UniqueName: \"kubernetes.io/projected/1ed38ab7-4bab-40e9-90d1-8532ad7ae572-kube-api-access-2jvkc\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:18.665358 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:18.665326    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5" event={"ID":"1ed38ab7-4bab-40e9-90d1-8532ad7ae572","Type":"ContainerDied","Data":"b06649e4aa28130af476e3b9c6faef2e71a08429b5215f6c92d1dbe8a0651294"}
Nov 01 23:30:18.665554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:18.665384    4837 scope.go:122] "RemoveContainer" containerID="02a72322ce0f7fca6fad83c3d3cbfc9dc4b7e16fd8aebfb5d339f52916699b81"
Nov 01 23:30:18.665554 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:18.665342    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"
Nov 01 23:30:18.813233 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:30:18.813153    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:30:18.818075 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:18.818051    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"]
Nov 01 23:30:18.864424 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:18.864394    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892930-w78l5"]
Nov 01 23:30:20.524436 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:20.524407    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" path="/var/lib/kubelet/pods/1ed38ab7-4bab-40e9-90d1-8532ad7ae572/volumes"
Nov 01 23:30:22.157898 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.157839    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v" podUID="ee0d91ca-5558-4f71-b10f-afa7e73f4387" containerName="main" containerID="cri-o://e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3" gracePeriod=45
Nov 01 23:30:22.260092 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.260029    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v" podUID="ee0d91ca-5558-4f71-b10f-afa7e73f4387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": dial tcp 10.134.0.216:8000: connect: connection refused"
Nov 01 23:30:22.409657 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.409601    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:30:22.537944 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.537915    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-model-cache\") pod \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") "
Nov 01 23:30:22.538156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.537974    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0d91ca-5558-4f71-b10f-afa7e73f4387-kube-api-access-srdjn\" (UniqueName: \"kubernetes.io/projected/ee0d91ca-5558-4f71-b10f-afa7e73f4387-kube-api-access-srdjn\") pod \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") "
Nov 01 23:30:22.538156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538004    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tmp-dir\") pod \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") "
Nov 01 23:30:22.538156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538023    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-dshm\") pod \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") "
Nov 01 23:30:22.538156 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538123    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tls-certs\") pod \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") "
Nov 01 23:30:22.538414 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538168    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-home\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-home\") pod \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\" (UID: \"ee0d91ca-5558-4f71-b10f-afa7e73f4387\") "
Nov 01 23:30:22.538414 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538253    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-model-cache" pod "ee0d91ca-5558-4f71-b10f-afa7e73f4387" (UID: "ee0d91ca-5558-4f71-b10f-afa7e73f4387"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:30:22.538414 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538306    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tmp-dir" pod "ee0d91ca-5558-4f71-b10f-afa7e73f4387" (UID: "ee0d91ca-5558-4f71-b10f-afa7e73f4387"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:30:22.538596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538494    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-home" pod "ee0d91ca-5558-4f71-b10f-afa7e73f4387" (UID: "ee0d91ca-5558-4f71-b10f-afa7e73f4387"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:30:22.538596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538514    4837 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-model-cache\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:22.538596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.538550    4837 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tmp-dir\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:22.540488 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.540464    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-dshm" pod "ee0d91ca-5558-4f71-b10f-afa7e73f4387" (UID: "ee0d91ca-5558-4f71-b10f-afa7e73f4387"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:30:22.540704 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.540679    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tls-certs" pod "ee0d91ca-5558-4f71-b10f-afa7e73f4387" (UID: "ee0d91ca-5558-4f71-b10f-afa7e73f4387"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:30:22.540783 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.540712    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0d91ca-5558-4f71-b10f-afa7e73f4387-kube-api-access-srdjn" pod "ee0d91ca-5558-4f71-b10f-afa7e73f4387" (UID: "ee0d91ca-5558-4f71-b10f-afa7e73f4387"). InnerVolumeSpecName "kube-api-access-srdjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:30:22.639288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.639259    4837 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee0d91ca-5558-4f71-b10f-afa7e73f4387-tls-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:22.639288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.639283    4837 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-home\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:22.639288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.639293    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srdjn\" (UniqueName: \"kubernetes.io/projected/ee0d91ca-5558-4f71-b10f-afa7e73f4387-kube-api-access-srdjn\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:22.639505 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.639304    4837 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee0d91ca-5558-4f71-b10f-afa7e73f4387-dshm\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:22.686225 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.686193    4837 generic.go:356] "Generic (PLEG): container finished" podID="ee0d91ca-5558-4f71-b10f-afa7e73f4387" containerID="e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3" exitCode=0
Nov 01 23:30:22.686387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.686238    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"
Nov 01 23:30:22.686387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.686274    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v" event={"ID":"ee0d91ca-5558-4f71-b10f-afa7e73f4387","Type":"ContainerDied","Data":"e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3"}
Nov 01 23:30:22.686387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.686317    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v" event={"ID":"ee0d91ca-5558-4f71-b10f-afa7e73f4387","Type":"ContainerDied","Data":"1f80cf293db515916b1142657626e27f4cdcc71b8ebce69f4de535719d6b7f19"}
Nov 01 23:30:22.686387 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.686335    4837 scope.go:122] "RemoveContainer" containerID="e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3"
Nov 01 23:30:22.695438 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.695417    4837 scope.go:122] "RemoveContainer" containerID="e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3"
Nov 01 23:30:22.695692 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:30:22.695675    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3\": container with ID starting with e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3 not found: ID does not exist" containerID="e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3"
Nov 01 23:30:22.695745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.695701    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3"} err="failed to get container status \"e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3\": rpc error: code = NotFound desc = could not find container \"e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3\": container with ID starting with e5b4bab830cfa81f726d9d3e653239d3ffdf3fc3fbee655e321c41be89229bb3 not found: ID does not exist"
Nov 01 23:30:22.840172 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.840139    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"]
Nov 01 23:30:22.844742 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:22.844719    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-863b60f0-kserve-7c5896fd97-zgw4v"]
Nov 01 23:30:24.523880 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:24.523851    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee0d91ca-5558-4f71-b10f-afa7e73f4387" path="/var/lib/kubelet/pods/ee0d91ca-5558-4f71-b10f-afa7e73f4387/volumes"
Nov 01 23:30:25.250579 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:25.250547    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:30:33.162697 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:33.162658    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:30:40.666247 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.666223    4837 scope.go:122] "RemoveContainer" containerID="3ab9700d0cb3e3b682c9315c3d41936ceb59bb7c7fe398be1cecc3c12a7f570d"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680191    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"]
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680589    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680600    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680617    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680623    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680635    4837 cpu_manager.go:414] "containerMap: removing container" podUID="ee0d91ca-5558-4f71-b10f-afa7e73f4387" containerName="main"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680640    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0d91ca-5558-4f71-b10f-afa7e73f4387" containerName="main"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680701    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0d91ca-5558-4f71-b10f-afa7e73f4387" containerName="main"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680710    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680718    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:40.681745 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.680726    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:40.684036 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.684020    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.735904 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.735877    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:30:40.736038 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.735984    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:30:40.736038 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.735997    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:30:40.782834 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.782805    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:30:40.784689 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.784669    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"]
Nov 01 23:30:40.804344 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.804315    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.804459 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.804358    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.804459 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.804438    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql24r\" (UniqueName: \"kubernetes.io/projected/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-kube-api-access-ql24r\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.905761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.905732    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.905761 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.905765    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.905994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.905782    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql24r\" (UniqueName: \"kubernetes.io/projected/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-kube-api-access-ql24r\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.906441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.906417    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.908161 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.908140    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.960004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.959935    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql24r\" (UniqueName: \"kubernetes.io/projected/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-kube-api-access-ql24r\") pod \"maas-api-revoke-sub-17147541-r8cmg\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:40.998901 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:40.998858    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:41.363019 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:41.362993    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"]
Nov 01 23:30:41.364435 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:30:41.364408    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2725d0e8_2875_4f4a_afb4_b0465c2ae1b7.slice/crio-5c676b1852ef4cea6852397fb0cd969613a299ebb3fb68972660bdef86b30182 WatchSource:0}: Error finding container 5c676b1852ef4cea6852397fb0cd969613a299ebb3fb68972660bdef86b30182: Status 404 returned error can't find the container with id 5c676b1852ef4cea6852397fb0cd969613a299ebb3fb68972660bdef86b30182
Nov 01 23:30:41.844476 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:41.844439    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg" event={"ID":"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7","Type":"ContainerStarted","Data":"0648b38eb1f1e21e356ab92632d79fc84b5a2ad4578c4327b8b42a64ddc0db7e"}
Nov 01 23:30:41.844881 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:41.844498    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg" event={"ID":"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7","Type":"ContainerStarted","Data":"5c676b1852ef4cea6852397fb0cd969613a299ebb3fb68972660bdef86b30182"}
Nov 01 23:30:41.964452 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:41.964401    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg" podStartSLOduration=1.964385478 podStartE2EDuration="1.964385478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:41.963217725 +0000 UTC m=+4023.552416787" watchObservedRunningTime="2026-11-01 23:30:41.964385478 +0000 UTC m=+4023.553584543"
Nov 01 23:30:42.850953 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:42.850913    4837 generic.go:356] "Generic (PLEG): container finished" podID="2725d0e8-2875-4f4a-afb4-b0465c2ae1b7" containerID="0648b38eb1f1e21e356ab92632d79fc84b5a2ad4578c4327b8b42a64ddc0db7e" exitCode=0
Nov 01 23:30:42.851373 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:42.850993    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg" event={"ID":"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7","Type":"ContainerDied","Data":"0648b38eb1f1e21e356ab92632d79fc84b5a2ad4578c4327b8b42a64ddc0db7e"}
Nov 01 23:30:43.995121 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:43.995094    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:44.137624 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.137541    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-kube-api-access-ql24r\" (UniqueName: \"kubernetes.io/projected/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-kube-api-access-ql24r\") pod \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") "
Nov 01 23:30:44.137812 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.137651    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-openshift-ca\") pod \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") "
Nov 01 23:30:44.137812 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.137673    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-xks-ca\") pod \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\" (UID: \"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7\") "
Nov 01 23:30:44.138095 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.138068    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-openshift-ca" pod "2725d0e8-2875-4f4a-afb4-b0465c2ae1b7" (UID: "2725d0e8-2875-4f4a-afb4-b0465c2ae1b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:30:44.139779 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.139756    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-kube-api-access-ql24r" pod "2725d0e8-2875-4f4a-afb4-b0465c2ae1b7" (UID: "2725d0e8-2875-4f4a-afb4-b0465c2ae1b7"). InnerVolumeSpecName "kube-api-access-ql24r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:30:44.139779 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.139766    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-xks-ca" pod "2725d0e8-2875-4f4a-afb4-b0465c2ae1b7" (UID: "2725d0e8-2875-4f4a-afb4-b0465c2ae1b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:30:44.238624 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.238595    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql24r\" (UniqueName: \"kubernetes.io/projected/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-kube-api-access-ql24r\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:44.238624 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.238621    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-openshift-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:44.238624 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.238633    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7-maas-api-xks-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:44.862237 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.862160    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"
Nov 01 23:30:44.862397 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.862156    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg" event={"ID":"2725d0e8-2875-4f4a-afb4-b0465c2ae1b7","Type":"ContainerDied","Data":"5c676b1852ef4cea6852397fb0cd969613a299ebb3fb68972660bdef86b30182"}
Nov 01 23:30:44.862397 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:44.862280    4837 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c676b1852ef4cea6852397fb0cd969613a299ebb3fb68972660bdef86b30182"
Nov 01 23:30:48.811073 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:30:48.811025    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:30:53.051425 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.051392    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"]
Nov 01 23:30:53.097948 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.097916    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-jxnxr"]
Nov 01 23:30:53.286408 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.286376    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:30:53.880211 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.880170    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"]
Nov 01 23:30:53.880596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.880576    4837 cpu_manager.go:414] "containerMap: removing container" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:53.880596 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.880591    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed38ab7-4bab-40e9-90d1-8532ad7ae572" containerName="cleanup"
Nov 01 23:30:53.880755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.880624    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2725d0e8-2875-4f4a-afb4-b0465c2ae1b7" containerName="revoke-keys"
Nov 01 23:30:53.880755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.880629    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2725d0e8-2875-4f4a-afb4-b0465c2ae1b7" containerName="revoke-keys"
Nov 01 23:30:53.880755 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.880690    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2725d0e8-2875-4f4a-afb4-b0465c2ae1b7" containerName="revoke-keys"
Nov 01 23:30:53.884286 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.884267    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:53.934584 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.934553    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d741b5c5-c6ba-4dad-8541-584bcd269406-openshift-service-ca\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:53.934728 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.934596    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-maas-api-tls\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:53.934728 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.934618    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-metrics-certs\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:53.934823 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.934770    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmt8j\" (UniqueName: \"kubernetes.io/projected/d741b5c5-c6ba-4dad-8541-584bcd269406-kube-api-access-jmt8j\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:53.979670 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.979642    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 23:30:53.979670 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.979654    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 23:30:53.979881 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.979668    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 23:30:53.979881 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.979726    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 23:30:53.979881 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.979807    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xp822\""
Nov 01 23:30:53.982032 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:53.982013    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"]
Nov 01 23:30:54.035666 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.035639    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmt8j\" (UniqueName: \"kubernetes.io/projected/d741b5c5-c6ba-4dad-8541-584bcd269406-kube-api-access-jmt8j\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.035843 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.035686    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d741b5c5-c6ba-4dad-8541-584bcd269406-openshift-service-ca\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.035843 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.035719    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-maas-api-tls\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.035843 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.035735    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-metrics-certs\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.036446 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.036416    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d741b5c5-c6ba-4dad-8541-584bcd269406-openshift-service-ca\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.038248 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.038224    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-maas-api-tls\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.038416 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.038393    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-metrics-certs\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.090295 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.090268    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmt8j\" (UniqueName: \"kubernetes.io/projected/d741b5c5-c6ba-4dad-8541-584bcd269406-kube-api-access-jmt8j\") pod \"maas-api-5bfbcdfbbb-wrhst\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.196300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.196269    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:54.524136 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.524035    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7" path="/var/lib/kubelet/pods/acfd9fce-f33e-4a43-a9a3-c6b64b81b7b7/volumes"
Nov 01 23:30:54.562947 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.562926    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"]
Nov 01 23:30:54.565093 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:30:54.565066    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd741b5c5_c6ba_4dad_8541_584bcd269406.slice/crio-b67355b0ba9afd34cb136e7c0d963709afd1d9b0015962b044975102d2cd5f4f WatchSource:0}: Error finding container b67355b0ba9afd34cb136e7c0d963709afd1d9b0015962b044975102d2cd5f4f: Status 404 returned error can't find the container with id b67355b0ba9afd34cb136e7c0d963709afd1d9b0015962b044975102d2cd5f4f
Nov 01 23:30:54.566354 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.566332    4837 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 23:30:54.916576 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:54.916549    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst" event={"ID":"d741b5c5-c6ba-4dad-8541-584bcd269406","Type":"ContainerStarted","Data":"b67355b0ba9afd34cb136e7c0d963709afd1d9b0015962b044975102d2cd5f4f"}
Nov 01 23:30:55.923358 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:55.923328    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst" event={"ID":"d741b5c5-c6ba-4dad-8541-584bcd269406","Type":"ContainerStarted","Data":"299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c"}
Nov 01 23:30:55.923807 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:55.923450    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:30:56.032830 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:30:56.032775    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst" podStartSLOduration=2.158048174 podStartE2EDuration="3.032761545s" totalImagesPullingTime="874.713371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:56.032712658 +0000 UTC m=+4037.621911719" watchObservedRunningTime="2026-11-01 23:30:56.032761545 +0000 UTC m=+4037.621960605"
Nov 01 23:31:01.934810 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:01.934783    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:31:02.749962 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.749928    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"]
Nov 01 23:31:02.753828 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.753807    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.825405 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.825369    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-metrics-certs\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.825604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.825436    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-openshift-service-ca\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.825604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.825569    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-maas-api-tls\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.825735 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.825609    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnx8\" (UniqueName: \"kubernetes.io/projected/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-kube-api-access-zpnx8\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.864117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.864087    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"]
Nov 01 23:31:02.926446 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.926416    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-metrics-certs\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.926676 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.926469    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-openshift-service-ca\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.926676 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.926509    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-maas-api-tls\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.926676 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.926523    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnx8\" (UniqueName: \"kubernetes.io/projected/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-kube-api-access-zpnx8\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.927279 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.927248    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-openshift-service-ca\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.929091 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.929062    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-metrics-certs\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.929211 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.929164    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-maas-api-tls\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:02.981140 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:02.981112    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnx8\" (UniqueName: \"kubernetes.io/projected/02ecca17-1d1a-4e4f-bc1e-ce25658df4c2-kube-api-access-zpnx8\") pod \"maas-api-58d749f5db-7z4n8\" (UID: \"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:03.066892 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:03.066815    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:03.435265 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:03.435241    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"]
Nov 01 23:31:03.436165 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:31:03.436137    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ecca17_1d1a_4e4f_bc1e_ce25658df4c2.slice/crio-77f8b57810bb4827f092063dc9a51f05924a826f7a973d92bdc4981484833eb0 WatchSource:0}: Error finding container 77f8b57810bb4827f092063dc9a51f05924a826f7a973d92bdc4981484833eb0: Status 404 returned error can't find the container with id 77f8b57810bb4827f092063dc9a51f05924a826f7a973d92bdc4981484833eb0
Nov 01 23:31:03.963916 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:03.963886    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8" event={"ID":"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2","Type":"ContainerStarted","Data":"77f8b57810bb4827f092063dc9a51f05924a826f7a973d92bdc4981484833eb0"}
Nov 01 23:31:04.970592 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:04.970560    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8" event={"ID":"02ecca17-1d1a-4e4f-bc1e-ce25658df4c2","Type":"ContainerStarted","Data":"6b3f60fdb4c9e61b764ab2afe0324b26437d502794a2e358ee177469cd152b9b"}
Nov 01 23:31:04.971078 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:04.970628    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:05.081600 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:05.081543    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8" podStartSLOduration=2.288996488 podStartE2EDuration="3.081529347s" totalImagesPullingTime="792.532859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:31:05.080438885 +0000 UTC m=+4046.669637945" watchObservedRunningTime="2026-11-01 23:31:05.081529347 +0000 UTC m=+4046.670728408"
Nov 01 23:31:10.981082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:10.981039    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d749f5db-7z4n8"
Nov 01 23:31:11.153282 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.153253    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"]
Nov 01 23:31:11.153519 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.153468    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst" podUID="d741b5c5-c6ba-4dad-8541-584bcd269406" containerName="maas-api" containerID="cri-o://299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c" gracePeriod=30
Nov 01 23:31:11.400869 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.400845    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:31:11.518320 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.518114    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d741b5c5-c6ba-4dad-8541-584bcd269406-kube-api-access-jmt8j\" (UniqueName: \"kubernetes.io/projected/d741b5c5-c6ba-4dad-8541-584bcd269406-kube-api-access-jmt8j\") pod \"d741b5c5-c6ba-4dad-8541-584bcd269406\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") "
Nov 01 23:31:11.518320 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.518177    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-maas-api-tls\") pod \"d741b5c5-c6ba-4dad-8541-584bcd269406\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") "
Nov 01 23:31:11.518320 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.518223    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d741b5c5-c6ba-4dad-8541-584bcd269406-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d741b5c5-c6ba-4dad-8541-584bcd269406-openshift-service-ca\") pod \"d741b5c5-c6ba-4dad-8541-584bcd269406\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") "
Nov 01 23:31:11.518320 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.518299    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-metrics-certs\") pod \"d741b5c5-c6ba-4dad-8541-584bcd269406\" (UID: \"d741b5c5-c6ba-4dad-8541-584bcd269406\") "
Nov 01 23:31:11.519154 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.519125    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d741b5c5-c6ba-4dad-8541-584bcd269406-openshift-service-ca" pod "d741b5c5-c6ba-4dad-8541-584bcd269406" (UID: "d741b5c5-c6ba-4dad-8541-584bcd269406"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:31:11.521143 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.521111    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-maas-api-tls" pod "d741b5c5-c6ba-4dad-8541-584bcd269406" (UID: "d741b5c5-c6ba-4dad-8541-584bcd269406"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:31:11.521361 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.521335    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d741b5c5-c6ba-4dad-8541-584bcd269406-kube-api-access-jmt8j" pod "d741b5c5-c6ba-4dad-8541-584bcd269406" (UID: "d741b5c5-c6ba-4dad-8541-584bcd269406"). InnerVolumeSpecName "kube-api-access-jmt8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:31:11.521772 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.521716    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-metrics-certs" pod "d741b5c5-c6ba-4dad-8541-584bcd269406" (UID: "d741b5c5-c6ba-4dad-8541-584bcd269406"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:31:11.619580 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.619544    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d741b5c5-c6ba-4dad-8541-584bcd269406-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:11.619580 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.619571    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:11.619580 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.619586    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmt8j\" (UniqueName: \"kubernetes.io/projected/d741b5c5-c6ba-4dad-8541-584bcd269406-kube-api-access-jmt8j\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:11.619903 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:11.619602    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d741b5c5-c6ba-4dad-8541-584bcd269406-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:12.006538 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.006509    4837 generic.go:356] "Generic (PLEG): container finished" podID="d741b5c5-c6ba-4dad-8541-584bcd269406" containerID="299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c" exitCode=0
Nov 01 23:31:12.007004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.006562    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"
Nov 01 23:31:12.007004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.006601    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst" event={"ID":"d741b5c5-c6ba-4dad-8541-584bcd269406","Type":"ContainerDied","Data":"299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c"}
Nov 01 23:31:12.007004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.006652    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst" event={"ID":"d741b5c5-c6ba-4dad-8541-584bcd269406","Type":"ContainerDied","Data":"b67355b0ba9afd34cb136e7c0d963709afd1d9b0015962b044975102d2cd5f4f"}
Nov 01 23:31:12.007004 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.006674    4837 scope.go:122] "RemoveContainer" containerID="299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c"
Nov 01 23:31:12.016420 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.016402    4837 scope.go:122] "RemoveContainer" containerID="299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c"
Nov 01 23:31:12.016701 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:31:12.016684    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c\": container with ID starting with 299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c not found: ID does not exist" containerID="299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c"
Nov 01 23:31:12.016788 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.016708    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c"} err="failed to get container status \"299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c\": rpc error: code = NotFound desc = could not find container \"299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c\": container with ID starting with 299c2aa5f4ba8c2fc95373e2c6d14b06c78657974e18c754fd95653cf001791c not found: ID does not exist"
Nov 01 23:31:12.163386 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.163354    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"]
Nov 01 23:31:12.209454 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.209422    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-wrhst"]
Nov 01 23:31:12.541382 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:12.541338    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d741b5c5-c6ba-4dad-8541-584bcd269406" path="/var/lib/kubelet/pods/d741b5c5-c6ba-4dad-8541-584bcd269406/volumes"
Nov 01 23:31:17.320135 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:17.320100    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:31:18.812299 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:31:18.812261    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:31:22.044975 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:22.044943    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"]
Nov 01 23:31:22.091344 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:22.091319    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-85tw2"]
Nov 01 23:31:22.523989 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:31:22.523958    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb259dc-57fa-4720-81ff-4a87fffafb01" path="/var/lib/kubelet/pods/ccb259dc-57fa-4720-81ff-4a87fffafb01/volumes"
Nov 01 23:31:48.812061 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:31:48.811962    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:32:04.342199 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:04.342168    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:32:14.992277 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:14.992242    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"]
Nov 01 23:32:14.992713 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:14.992698    4837 cpu_manager.go:414] "containerMap: removing container" podUID="d741b5c5-c6ba-4dad-8541-584bcd269406" containerName="maas-api"
Nov 01 23:32:14.992760 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:14.992714    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d741b5c5-c6ba-4dad-8541-584bcd269406" containerName="maas-api"
Nov 01 23:32:14.992811 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:14.992802    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="d741b5c5-c6ba-4dad-8541-584bcd269406" containerName="maas-api"
Nov 01 23:32:14.995956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:14.995937    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.045375 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.045350    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-2474f7-serving-cert\""
Nov 01 23:32:15.093699 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.093672    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"]
Nov 01 23:32:15.169493 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.169461    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxhg\" (UniqueName: \"kubernetes.io/projected/2e9cb989-a2e3-4102-bb70-7365b76b38ac-kube-api-access-wqxhg\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.169667 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.169530    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-metrics-certs\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.169667 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.169617    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.169667 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.169645    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9cb989-a2e3-4102-bb70-7365b76b38ac-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.271184 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.271102    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxhg\" (UniqueName: \"kubernetes.io/projected/2e9cb989-a2e3-4102-bb70-7365b76b38ac-kube-api-access-wqxhg\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.271184 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.271159    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-metrics-certs\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.271184 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.271183    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.271497 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.271199    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9cb989-a2e3-4102-bb70-7365b76b38ac-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.271879 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.271858    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9cb989-a2e3-4102-bb70-7365b76b38ac-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.274117 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.274097    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-metrics-certs\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.274287 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.274264    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.325344 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.325312    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxhg\" (UniqueName: \"kubernetes.io/projected/2e9cb989-a2e3-4102-bb70-7365b76b38ac-kube-api-access-wqxhg\") pod \"maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.608900 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.608808    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:15.974436 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:15.974411    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"]
Nov 01 23:32:15.975556 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:32:15.975529    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9cb989_a2e3_4102_bb70_7365b76b38ac.slice/crio-8f52aee7425cfabc5d7abe202b2b74c57196c182a915739735e3dfc1dd0db492 WatchSource:0}: Error finding container 8f52aee7425cfabc5d7abe202b2b74c57196c182a915739735e3dfc1dd0db492: Status 404 returned error can't find the container with id 8f52aee7425cfabc5d7abe202b2b74c57196c182a915739735e3dfc1dd0db492
Nov 01 23:32:16.353507 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:16.353424    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh" event={"ID":"2e9cb989-a2e3-4102-bb70-7365b76b38ac","Type":"ContainerStarted","Data":"8f52aee7425cfabc5d7abe202b2b74c57196c182a915739735e3dfc1dd0db492"}
Nov 01 23:32:17.360613 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:17.360574    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh" event={"ID":"2e9cb989-a2e3-4102-bb70-7365b76b38ac","Type":"ContainerStarted","Data":"f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b"}
Nov 01 23:32:17.361103 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:17.360705    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:17.478108 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:17.478033    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh" podStartSLOduration=2.602180918 podStartE2EDuration="3.478020134s" totalImagesPullingTime="875.839216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:32:17.475995684 +0000 UTC m=+4119.065194751" watchObservedRunningTime="2026-11-01 23:32:17.478020134 +0000 UTC m=+4119.067219194"
Nov 01 23:32:18.812872 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:32:18.812839    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:32:22.776752 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:22.776720    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:32:23.371288 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:23.371260    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:32:40.707372 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:40.707338    4837 scope.go:122] "RemoveContainer" containerID="7abc8d6131f8b32c5e3719a7b600ea773cbe2c2bd81df1243e3e0eee35023df1"
Nov 01 23:32:40.715803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:32:40.715784    4837 scope.go:122] "RemoveContainer" containerID="a86576a9047b7f60ed44176e2ef43f45ce30b999900908eab6bfa34faf5b8430"
Nov 01 23:32:48.811870 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:32:48.811823    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:33:18.811464 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:33:18.811378    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:33:39.432089 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:33:39.432036    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:33:39.438266 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:33:39.438244    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-m7tdl_b330b73a-8364-478a-9589-1b24674fb30e/console-operator/2.log"
Nov 01 23:33:48.814487 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:33:48.814451    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:34:16.591189 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.591152    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"]
Nov 01 23:34:16.591781 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.591414    4837 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh" podUID="2e9cb989-a2e3-4102-bb70-7365b76b38ac" containerName="maas-api" containerID="cri-o://f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b" gracePeriod=30
Nov 01 23:34:16.859994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.859970    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:34:16.928260 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.928231    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-metrics-certs\") pod \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") "
Nov 01 23:34:16.928441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.928296    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-maas-api-tls\") pod \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") "
Nov 01 23:34:16.928441 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.928382    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9cb989-a2e3-4102-bb70-7365b76b38ac-kube-api-access-wqxhg\" (UniqueName: \"kubernetes.io/projected/2e9cb989-a2e3-4102-bb70-7365b76b38ac-kube-api-access-wqxhg\") pod \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") "
Nov 01 23:34:16.928548 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.928534    4837 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9cb989-a2e3-4102-bb70-7365b76b38ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9cb989-a2e3-4102-bb70-7365b76b38ac-openshift-service-ca\") pod \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\" (UID: \"2e9cb989-a2e3-4102-bb70-7365b76b38ac\") "
Nov 01 23:34:16.928958 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.928908    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9cb989-a2e3-4102-bb70-7365b76b38ac-openshift-service-ca" pod "2e9cb989-a2e3-4102-bb70-7365b76b38ac" (UID: "2e9cb989-a2e3-4102-bb70-7365b76b38ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:34:16.930371 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.930346    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-maas-api-tls" pod "2e9cb989-a2e3-4102-bb70-7365b76b38ac" (UID: "2e9cb989-a2e3-4102-bb70-7365b76b38ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:34:16.930492 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.930393    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-metrics-certs" pod "2e9cb989-a2e3-4102-bb70-7365b76b38ac" (UID: "2e9cb989-a2e3-4102-bb70-7365b76b38ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:34:16.930492 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.930479    4837 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9cb989-a2e3-4102-bb70-7365b76b38ac-kube-api-access-wqxhg" pod "2e9cb989-a2e3-4102-bb70-7365b76b38ac" (UID: "2e9cb989-a2e3-4102-bb70-7365b76b38ac"). InnerVolumeSpecName "kube-api-access-wqxhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:34:16.958968 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.958939    4837 generic.go:356] "Generic (PLEG): container finished" podID="2e9cb989-a2e3-4102-bb70-7365b76b38ac" containerID="f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b" exitCode=0
Nov 01 23:34:16.959119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.958989    4837 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"
Nov 01 23:34:16.959119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.959025    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh" event={"ID":"2e9cb989-a2e3-4102-bb70-7365b76b38ac","Type":"ContainerDied","Data":"f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b"}
Nov 01 23:34:16.959119 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.959112    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh" event={"ID":"2e9cb989-a2e3-4102-bb70-7365b76b38ac","Type":"ContainerDied","Data":"8f52aee7425cfabc5d7abe202b2b74c57196c182a915739735e3dfc1dd0db492"}
Nov 01 23:34:16.959232 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.959144    4837 scope.go:122] "RemoveContainer" containerID="f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b"
Nov 01 23:34:16.969487 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.969463    4837 scope.go:122] "RemoveContainer" containerID="f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b"
Nov 01 23:34:16.972417 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:34:16.972391    4837 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b\": container with ID starting with f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b not found: ID does not exist" containerID="f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b"
Nov 01 23:34:16.972703 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:16.972675    4837 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b"} err="failed to get container status \"f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b\": rpc error: code = NotFound desc = could not find container \"f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b\": container with ID starting with f4d8b2aca9aae0af9c958a2a352ad9002eeec2ba87c5af363c86ace2693b179b not found: ID does not exist"
Nov 01 23:34:17.030127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:17.030093    4837 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9cb989-a2e3-4102-bb70-7365b76b38ac-openshift-service-ca\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:17.030127 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:17.030127    4837 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-metrics-certs\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:17.030352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:17.030139    4837 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9cb989-a2e3-4102-bb70-7365b76b38ac-maas-api-tls\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:17.030352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:17.030152    4837 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxhg\" (UniqueName: \"kubernetes.io/projected/2e9cb989-a2e3-4102-bb70-7365b76b38ac-kube-api-access-wqxhg\") on node \"ip-10-0-129-77.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:17.120597 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:17.120509    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"]
Nov 01 23:34:17.130011 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:17.129975    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-2474f7-868f556678-7p7bh"]
Nov 01 23:34:18.523419 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:18.523379    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e9cb989-a2e3-4102-bb70-7365b76b38ac" path="/var/lib/kubelet/pods/2e9cb989-a2e3-4102-bb70-7365b76b38ac/volumes"
Nov 01 23:34:18.812772 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:34:18.812684    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:34:21.160412 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:21.160371    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:34:34.556531 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:34.556500    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:34:44.052671 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:44.052639    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:34:48.813214 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:34:48.813132    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:34:54.042994 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:54.042954    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"]
Nov 01 23:34:54.089340 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:54.089215    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gwvcz"]
Nov 01 23:34:54.523588 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:34:54.523559    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f459ca-bf39-436c-8420-7a3c08bb46df" path="/var/lib/kubelet/pods/68f459ca-bf39-436c-8420-7a3c08bb46df/volumes"
Nov 01 23:35:04.960687 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:04.960648    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:35:18.811619 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:35:18.811581    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:35:37.247828 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:37.247792    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhprg"]
Nov 01 23:35:38.748275 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:38.748236    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pgdbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:35:38.748935 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:38.748900    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pgdbs" podUID="010f17eb-e378-4b34-9924-55aeba33558a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:35:38.789483 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:38.789450    4837 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pp5tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:35:38.789795 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:38.789756    4837 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pp5tz" podUID="cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:35:45.043803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:45.043769    4837 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"]
Nov 01 23:35:45.090211 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:45.090183    4837 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17147541-r8cmg"]
Nov 01 23:35:46.523551 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:46.523514    4837 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2725d0e8-2875-4f4a-afb4-b0465c2ae1b7" path="/var/lib/kubelet/pods/2725d0e8-2875-4f4a-afb4-b0465c2ae1b7/volumes"
Nov 01 23:35:46.796211 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:46.796139    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-847995f698-ll4ng_7966306e-e19c-4423-a82f-5827ce9d2d63/manager/0.log"
Nov 01 23:35:47.148223 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:47.148145    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wt2xw_4940c23d-3994-46bb-bea1-7303c8a50498/manager/0.log"
Nov 01 23:35:47.941118 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:47.941086    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c9f79b66c-hjmv4_b8451fe9-c49f-48fe-877c-d86a2a68509c/manager/0.log"
Nov 01 23:35:48.629604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:48.629572    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-26jmv_b5861219-3f6b-475c-810b-168da625ec9a/manager/0.log"
Nov 01 23:35:48.815578 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:35:48.815386    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:35:49.593376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:49.593347    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58d749f5db-7z4n8_02ecca17-1d1a-4e4f-bc1e-ce25658df4c2/maas-api/0.log"
Nov 01 23:35:53.624502 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:53.624477    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lhprg_4c058156-64d8-4831-9056-3d62bc7f0bff/limitador/0.log"
Nov 01 23:35:54.848377 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:54.848344    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l5dx5_aadcdb38-ad92-4d97-a143-7753934cd55b/discovery/0.log"
Nov 01 23:35:55.063893 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:55.063850    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b78b6bf6-drlpm_cb8723c9-c41d-483b-9805-df01a4a041d3/kube-auth-proxy/0.log"
Nov 01 23:35:56.704352 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:56.704322    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-jkb5h_1cd9e0f1-2298-4a4b-8946-5327b4787729/istio-proxy/0.log"
Nov 01 23:35:57.202604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:57.202576    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-pgdbs_010f17eb-e378-4b34-9924-55aeba33558a/istio-proxy/0.log"
Nov 01 23:35:57.410481 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:57.410450    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-pp5tz_cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee/istio-proxy/0.log"
Nov 01 23:35:57.632951 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:57.632879    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-zvlcq_43410c5e-e1c9-4389-b1e0-c12e99842a84/istio-proxy/0.log"
Nov 01 23:35:58.076343 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:58.076317    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-5h4j6_58a7115a-269b-499d-8dd7-5e92d093bb8e/payload-pre-processing/0.log"
Nov 01 23:35:58.280604 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:58.280572    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-xzwhj_2c8cad21-8361-4f7e-b87e-17359415a82e/payload-processing/0.log"
Nov 01 23:35:58.496526 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:58.496499    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56cdb546f-44l9r_234fcee4-4ad0-4144-8b1b-7f16e714a08f/router/0.log"
Nov 01 23:35:59.275170 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:59.275139    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v_22a9dcd7-fec2-4d08-93c9-09ff3ec499a0/main/0.log"
Nov 01 23:35:59.920743 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:35:59.920715    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4jbsp_7ea44462-9d6a-42ec-ae71-4f8b3567f7b1/main/0.log"
Nov 01 23:36:00.408678 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:00.408604    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-t294z_ef383e25-1da9-4495-85d4-e22fbcd68022/main/0.log"
Nov 01 23:36:00.688679 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:00.688651    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lq565_87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9/main/0.log"
Nov 01 23:36:10.619376 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:10.619345    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-847995f698-ll4ng_7966306e-e19c-4423-a82f-5827ce9d2d63/manager/0.log"
Nov 01 23:36:10.990504 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:10.990473    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wt2xw_4940c23d-3994-46bb-bea1-7303c8a50498/manager/0.log"
Nov 01 23:36:11.709577 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:11.709538    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c9f79b66c-hjmv4_b8451fe9-c49f-48fe-877c-d86a2a68509c/manager/0.log"
Nov 01 23:36:12.161976 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:12.161889    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-26jmv_b5861219-3f6b-475c-810b-168da625ec9a/manager/0.log"
Nov 01 23:36:12.942544 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:12.942511    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58d749f5db-7z4n8_02ecca17-1d1a-4e4f-bc1e-ce25658df4c2/maas-api/0.log"
Nov 01 23:36:16.570257 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:16.570226    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lhprg_4c058156-64d8-4831-9056-3d62bc7f0bff/limitador/0.log"
Nov 01 23:36:17.603708 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:17.603676    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l5dx5_aadcdb38-ad92-4d97-a143-7753934cd55b/discovery/0.log"
Nov 01 23:36:17.850826 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:17.850745    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b78b6bf6-drlpm_cb8723c9-c41d-483b-9805-df01a4a041d3/kube-auth-proxy/0.log"
Nov 01 23:36:18.812716 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:36:18.812681    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:36:19.477238 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:19.477209    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-jkb5h_1cd9e0f1-2298-4a4b-8946-5327b4787729/istio-proxy/0.log"
Nov 01 23:36:19.951331 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:19.951304    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-pgdbs_010f17eb-e378-4b34-9924-55aeba33558a/istio-proxy/0.log"
Nov 01 23:36:20.173082 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:20.173053    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-pp5tz_cdfea7d7-2f9c-41c8-9dbb-b32edf6b67ee/istio-proxy/0.log"
Nov 01 23:36:20.386472 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:20.386374    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-zvlcq_43410c5e-e1c9-4389-b1e0-c12e99842a84/istio-proxy/0.log"
Nov 01 23:36:20.824556 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:20.824528    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-5h4j6_58a7115a-269b-499d-8dd7-5e92d093bb8e/payload-pre-processing/0.log"
Nov 01 23:36:21.041177 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:21.041147    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-xzwhj_2c8cad21-8361-4f7e-b87e-17359415a82e/payload-processing/0.log"
Nov 01 23:36:21.258803 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:21.258770    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56cdb546f-44l9r_234fcee4-4ad0-4144-8b1b-7f16e714a08f/router/0.log"
Nov 01 23:36:21.812626 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:21.812597    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9nz9v_22a9dcd7-fec2-4d08-93c9-09ff3ec499a0/main/0.log"
Nov 01 23:36:22.485020 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:22.484983    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4jbsp_7ea44462-9d6a-42ec-ae71-4f8b3567f7b1/main/0.log"
Nov 01 23:36:22.960062 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:22.960008    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-t294z_ef383e25-1da9-4495-85d4-e22fbcd68022/main/0.log"
Nov 01 23:36:23.225300 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:23.225204    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lq565_87b00ef5-41fb-4dbd-834e-c2cb76f1e6f9/main/0.log"
Nov 01 23:36:40.769087 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:40.769036    4837 scope.go:122] "RemoveContainer" containerID="d7681c44053c39c37c8cde7f21a93d0d5547da5e89fc25f2c920609ecd990fc0"
Nov 01 23:36:48.811928 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:36:48.811892    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:36:49.888671 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:49.888641    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c9f79b66c-hjmv4_b8451fe9-c49f-48fe-877c-d86a2a68509c/manager/0.log"
Nov 01 23:36:50.429638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:50.429602    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c9f79b66c-hjmv4_b8451fe9-c49f-48fe-877c-d86a2a68509c/manager/0.log"
Nov 01 23:36:51.125741 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:51.125714    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-xzwhj_2c8cad21-8361-4f7e-b87e-17359415a82e/payload-processing/0.log"
Nov 01 23:36:51.346798 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:36:51.346765    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-xzwhj_2c8cad21-8361-4f7e-b87e-17359415a82e/payload-processing/0.log"
Nov 01 23:37:18.811835 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:37:18.811803    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:37:26.033333 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:26.033302    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c9f79b66c-hjmv4_b8451fe9-c49f-48fe-877c-d86a2a68509c/manager/0.log"
Nov 01 23:37:26.605321 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:26.605273    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c9f79b66c-hjmv4_b8451fe9-c49f-48fe-877c-d86a2a68509c/manager/0.log"
Nov 01 23:37:27.290698 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:27.290669    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-xzwhj_2c8cad21-8361-4f7e-b87e-17359415a82e/payload-processing/0.log"
Nov 01 23:37:27.499986 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:27.499959    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-xzwhj_2c8cad21-8361-4f7e-b87e-17359415a82e/payload-processing/0.log"
Nov 01 23:37:36.959455 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:36.959425    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-njrpj_c3e0e60c-b299-48da-86b6-3cf0dfdfa238/global-pull-secret-syncer/0.log"
Nov 01 23:37:38.101963 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:38.101935    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gwk8z_1878f36c-168f-490a-bfef-bbe0f477bf88/connection-checker/0.log"
Nov 01 23:37:38.507192 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:38.507163    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ktgdl_d66bf75c-5104-4918-b1f5-48fc89bf7f73/connection-checker/0.log"
Nov 01 23:37:38.915615 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:38.915537    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rhtp4_dc731cae-588c-4d82-850e-1c3c3f954833/connection-checker/0.log"
Nov 01 23:37:39.809555 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:39.809516    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hnmh2_2ee57462-5ba3-4cfe-a233-13c046a7e797/konnectivity-agent/0.log"
Nov 01 23:37:40.788201 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:40.788169    4837 scope.go:122] "RemoveContainer" containerID="0648b38eb1f1e21e356ab92632d79fc84b5a2ad4578c4327b8b42a64ddc0db7e"
Nov 01 23:37:40.796743 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:40.796720    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-77.us-west-2.compute.internal_0dcabca36fe7386806bacef405dbcfa5/haproxy/0.log"
Nov 01 23:37:46.932083 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:46.932033    4837 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"]
Nov 01 23:37:46.932487 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:46.932464    4837 cpu_manager.go:414] "containerMap: removing container" podUID="2e9cb989-a2e3-4102-bb70-7365b76b38ac" containerName="maas-api"
Nov 01 23:37:46.932487 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:46.932474    4837 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9cb989-a2e3-4102-bb70-7365b76b38ac" containerName="maas-api"
Nov 01 23:37:46.932561 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:46.932544    4837 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9cb989-a2e3-4102-bb70-7365b76b38ac" containerName="maas-api"
Nov 01 23:37:46.935638 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:46.935618    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:46.987837 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:46.987800    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wg4hr\"/\"openshift-service-ca.crt\""
Nov 01 23:37:46.990037 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:46.990013    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"]
Nov 01 23:37:47.034381 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.034354    4837 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wg4hr\"/\"default-dockercfg-76nmg\""
Nov 01 23:37:47.034529 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.034402    4837 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wg4hr\"/\"kube-root-ca.crt\""
Nov 01 23:37:47.072840 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.072808    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-sys\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.073017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.072854    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-proc\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.073017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.072906    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-lib-modules\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.073017 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.072927    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqxv\" (UniqueName: \"kubernetes.io/projected/c4f4597d-594a-4d85-a0e7-c3f71cca8980-kube-api-access-lsqxv\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.073165 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.073013    4837 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-podres\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173742 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173707    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-proc\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173742 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173748    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-lib-modules\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173769    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqxv\" (UniqueName: \"kubernetes.io/projected/c4f4597d-594a-4d85-a0e7-c3f71cca8980-kube-api-access-lsqxv\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173808    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-podres\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173838    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-proc\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173854    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-lib-modules\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173864    4837 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-sys\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173910    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-sys\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.173956 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.173917    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4f4597d-594a-4d85-a0e7-c3f71cca8980-podres\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.226664 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.226591    4837 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqxv\" (UniqueName: \"kubernetes.io/projected/c4f4597d-594a-4d85-a0e7-c3f71cca8980-kube-api-access-lsqxv\") pod \"perf-node-gather-daemonset-qmp7n\" (UID: \"c4f4597d-594a-4d85-a0e7-c3f71cca8980\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.248569 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.248536    4837 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:47.614142 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:47.614116    4837 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"]
Nov 01 23:37:47.616569 ip-10-0-129-77 kubenswrapper[4837]: W1101 23:37:47.616540    4837 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc4f4597d_594a_4d85_a0e7_c3f71cca8980.slice/crio-a423c5dc883c7ff3ccff90113a2fe016b5de580d16c3a0c3601d5e8d1eb7ccdb WatchSource:0}: Error finding container a423c5dc883c7ff3ccff90113a2fe016b5de580d16c3a0c3601d5e8d1eb7ccdb: Status 404 returned error can't find the container with id a423c5dc883c7ff3ccff90113a2fe016b5de580d16c3a0c3601d5e8d1eb7ccdb
Nov 01 23:37:48.076785 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:48.076747    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n" event={"ID":"c4f4597d-594a-4d85-a0e7-c3f71cca8980","Type":"ContainerStarted","Data":"a65e179b838f45f07287b389c7e70af5807bb96808530104c5bc1a08310f6570"}
Nov 01 23:37:48.077249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:48.076800    4837 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n" event={"ID":"c4f4597d-594a-4d85-a0e7-c3f71cca8980","Type":"ContainerStarted","Data":"a423c5dc883c7ff3ccff90113a2fe016b5de580d16c3a0c3601d5e8d1eb7ccdb"}
Nov 01 23:37:48.077249 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:48.076878    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:48.178299 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:48.178274    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lhprg_4c058156-64d8-4831-9056-3d62bc7f0bff/limitador/0.log"
Nov 01 23:37:48.183540 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:48.183500    4837 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n" podStartSLOduration=2.18348638 podStartE2EDuration="2.18348638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 23:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:37:48.182602745 +0000 UTC m=+4449.771801807" watchObservedRunningTime="2026-11-01 23:37:48.18348638 +0000 UTC m=+4449.772685440"
Nov 01 23:37:48.813148 ip-10-0-129-77 kubenswrapper[4837]: E1101 23:37:48.813115    4837 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-h84pf" containerName="extractor"
Nov 01 23:37:54.091887 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:54.091854    4837 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-qmp7n"
Nov 01 23:37:54.216235 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:54.216202    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ss6wx_d41ea86a-a166-44b7-a8da-2a7512c9e786/cluster-monitoring-operator/0.log"
Nov 01 23:37:59.180271 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:59.180240    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6djl_5d16f06c-40ec-43c7-8a0a-8959bb077e10/node-exporter/0.log"
Nov 01 23:37:59.596937 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:37:59.596844    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6djl_5d16f06c-40ec-43c7-8a0a-8959bb077e10/kube-rbac-proxy/0.log"
Nov 01 23:38:00.016897 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:38:00.016871    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6djl_5d16f06c-40ec-43c7-8a0a-8959bb077e10/init-textfile/0.log"
Nov 01 23:38:09.802622 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:38:09.802588    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l5dx5_aadcdb38-ad92-4d97-a143-7753934cd55b/discovery/0.log"
Nov 01 23:38:09.907333 ip-10-0-129-77 kubenswrapper[4837]: I1101 23:38:09.907291    4837 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b78b6bf6-drlpm_cb8723c9-c41d-483b-9805-df01a4a041d3/kube-auth-proxy/0.log"